{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "d9ba5257",
   "metadata": {
    "origin_pos": 0
   },
   "source": [
    "# 梯度下降\n",
    ":label:`sec_gd`\n",
    "\n",
    "尽管*梯度下降*（gradient descent）很少直接用于深度学习，\n",
    "但了解它是理解下一节随机梯度下降算法的关键。\n",
    "例如，由于学习率过大，优化问题可能会发散，这种现象早已在梯度下降中出现。\n",
    "同样地，*预处理*（preconditioning）是梯度下降中的一种常用技术，\n",
    "还被沿用到更高级的算法中。\n",
    "让我们从简单的一维梯度下降开始。\n",
    "\n",
    "## 一维梯度下降\n",
    "\n",
    "为什么梯度下降算法可以优化目标函数？\n",
    "一维中的梯度下降给我们很好的启发。\n",
    "考虑一类连续可微实值函数$f: \\mathbb{R} \\rightarrow \\mathbb{R}$，\n",
    "利用泰勒展开，我们可以得到\n",
    "\n",
    "$$f(x + \\epsilon) = f(x) + \\epsilon f'(x) + \\mathcal{O}(\\epsilon^2).$$\n",
    ":eqlabel:`gd-taylor`\n",
    "\n",
    "即在一阶近似中，$f(x+\\epsilon)$可通过$x$处的函数值$f(x)$和一阶导数$f'(x)$得出。\n",
    "我们可以假设在负梯度方向上移动的$\\epsilon$会减少$f$。\n",
    "为了简单起见，我们选择固定步长$\\eta > 0$，然后取$\\epsilon = -\\eta f'(x)$。\n",
    "将其代入泰勒展开式我们可以得到\n",
    "\n",
    "$$f(x - \\eta f'(x)) = f(x) - \\eta f'^2(x) + \\mathcal{O}(\\eta^2 f'^2(x)).$$\n",
    ":eqlabel:`gd-taylor-2`\n",
    "\n",
    "如果其导数$f'(x) \\neq 0$没有消失，我们就能继续展开，这是因为$\\eta f'^2(x)>0$。\n",
    "此外，我们总是可以令$\\eta$小到足以使高阶项变得不相关。\n",
    "因此，\n",
    "\n",
    "$$f(x - \\eta f'(x)) \\lessapprox f(x).$$\n",
    "\n",
    "这意味着，如果我们使用\n",
    "\n",
    "$$x \\leftarrow x - \\eta f'(x)$$\n",
    "\n",
    "来迭代$x$，函数$f(x)$的值可能会下降。\n",
    "因此，在梯度下降中，我们首先选择初始值$x$和常数$\\eta > 0$，\n",
    "然后使用它们连续迭代$x$，直到停止条件达成。\n",
    "例如，当梯度$|f'(x)|$的幅度足够小或迭代次数达到某个值时。\n",
    "\n",
    "下面我们来展示如何实现梯度下降。为了简单起见，我们选用目标函数$f(x)=x^2$。\n",
    "尽管我们知道$x=0$时$f(x)$能取得最小值，\n",
    "但我们仍然使用这个简单的函数来观察$x$的变化。\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "7b783f74",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-08-18T07:01:23.083492Z",
     "iopub.status.busy": "2023-08-18T07:01:23.082697Z",
     "iopub.status.idle": "2023-08-18T07:01:26.147110Z",
     "shell.execute_reply": "2023-08-18T07:01:26.145926Z"
    },
    "origin_pos": 2,
    "tab": [
     "pytorch"
    ]
   },
   "outputs": [],
   "source": [
    "%matplotlib inline\n",
    "import numpy as np\n",
    "import torch\n",
    "from d2l import torch as d2l"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "b544b8ce",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-08-18T07:01:26.152743Z",
     "iopub.status.busy": "2023-08-18T07:01:26.151955Z",
     "iopub.status.idle": "2023-08-18T07:01:26.158133Z",
     "shell.execute_reply": "2023-08-18T07:01:26.157003Z"
    },
    "origin_pos": 5,
    "tab": [
     "pytorch"
    ]
   },
   "outputs": [],
   "source": [
    "def f(x):  # 目标函数\n",
    "    return x ** 2\n",
    "\n",
    "def f_grad(x):  # 目标函数的梯度(导数)\n",
    "    return 2 * x"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "784941de",
   "metadata": {
    "origin_pos": 6
   },
   "source": [
    "接下来，我们使用$x=10$作为初始值，并假设$\\eta=0.2$。\n",
    "使用梯度下降法迭代$x$共10次，我们可以看到，$x$的值最终将接近最优解。\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "3674fea9",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-08-18T07:01:26.163008Z",
     "iopub.status.busy": "2023-08-18T07:01:26.162385Z",
     "iopub.status.idle": "2023-08-18T07:01:26.170126Z",
     "shell.execute_reply": "2023-08-18T07:01:26.169003Z"
    },
    "origin_pos": 7,
    "tab": [
     "pytorch"
    ]
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "epoch 10, x: 0.060466\n"
     ]
    }
   ],
   "source": [
    "def gd(eta, f_grad):\n",
    "    x = 10.0\n",
    "    results = [x]\n",
    "    for i in range(10):\n",
    "        x -= eta * f_grad(x)\n",
    "        results.append(float(x))\n",
    "    print(f'epoch 10, x: {x:f}')\n",
    "    return results\n",
    "\n",
    "results = gd(0.2, f_grad)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a005e799",
   "metadata": {
    "origin_pos": 9
   },
   "source": [
    "对进行$x$优化的过程可以绘制如下。\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "56e7d8fb",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-08-18T07:01:26.174775Z",
     "iopub.status.busy": "2023-08-18T07:01:26.174138Z",
     "iopub.status.idle": "2023-08-18T07:01:26.480544Z",
     "shell.execute_reply": "2023-08-18T07:01:26.479162Z"
    },
    "origin_pos": 10,
    "tab": [
     "pytorch"
    ]
   },
   "outputs": [
    {
     "data": {
      "image/svg+xml": [
       "<?xml version=\"1.0\" encoding=\"utf-8\" standalone=\"no\"?>\n",
       "<!DOCTYPE svg PUBLIC \"-//W3C//DTD SVG 1.1//EN\"\n",
       "  \"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd\">\n",
       "<svg xmlns:xlink=\"http://www.w3.org/1999/xlink\" width=\"249.465625pt\" height=\"180.65625pt\" viewBox=\"0 0 249.465625 180.65625\" xmlns=\"http://www.w3.org/2000/svg\" version=\"1.1\">\n",
       " <metadata>\n",
       "  <rdf:RDF xmlns:dc=\"http://purl.org/dc/elements/1.1/\" xmlns:cc=\"http://creativecommons.org/ns#\" xmlns:rdf=\"http://www.w3.org/1999/02/22-rdf-syntax-ns#\">\n",
       "   <cc:Work>\n",
       "    <dc:type rdf:resource=\"http://purl.org/dc/dcmitype/StillImage\"/>\n",
       "    <dc:date>2023-08-18T07:01:26.426728</dc:date>\n",
       "    <dc:format>image/svg+xml</dc:format>\n",
       "    <dc:creator>\n",
       "     <cc:Agent>\n",
       "      <dc:title>Matplotlib v3.5.1, https://matplotlib.org/</dc:title>\n",
       "     </cc:Agent>\n",
       "    </dc:creator>\n",
       "   </cc:Work>\n",
       "  </rdf:RDF>\n",
       " </metadata>\n",
       " <defs>\n",
       "  <style type=\"text/css\">*{stroke-linejoin: round; stroke-linecap: butt}</style>\n",
       " </defs>\n",
       " <g id=\"figure_1\">\n",
       "  <g id=\"patch_1\">\n",
       "   <path d=\"M 0 180.65625 \n",
       "L 249.465625 180.65625 \n",
       "L 249.465625 0 \n",
       "L 0 0 \n",
       "L 0 180.65625 \n",
       "z\n",
       "\" style=\"fill: none\"/>\n",
       "  </g>\n",
       "  <g id=\"axes_1\">\n",
       "   <g id=\"patch_2\">\n",
       "    <path d=\"M 46.965625 143.1 \n",
       "L 242.265625 143.1 \n",
       "L 242.265625 7.2 \n",
       "L 46.965625 7.2 \n",
       "z\n",
       "\" style=\"fill: #ffffff\"/>\n",
       "   </g>\n",
       "   <g id=\"matplotlib.axis_1\">\n",
       "    <g id=\"xtick_1\">\n",
       "     <g id=\"line2d_1\">\n",
       "      <path d=\"M 55.842898 143.1 \n",
       "L 55.842898 7.2 \n",
       "\" clip-path=\"url(#p62bcb874e4)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_2\">\n",
       "      <defs>\n",
       "       <path id=\"mcbc85dced4\" d=\"M 0 0 \n",
       "L 0 3.5 \n",
       "\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </defs>\n",
       "      <g>\n",
       "       <use xlink:href=\"#mcbc85dced4\" x=\"55.842898\" y=\"143.1\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_1\">\n",
       "      <!-- −10 -->\n",
       "      <g transform=\"translate(45.290554 157.698438)scale(0.1 -0.1)\">\n",
       "       <defs>\n",
       "        <path id=\"DejaVuSans-2212\" d=\"M 678 2272 \n",
       "L 4684 2272 \n",
       "L 4684 1741 \n",
       "L 678 1741 \n",
       "L 678 2272 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "        <path id=\"DejaVuSans-31\" d=\"M 794 531 \n",
       "L 1825 531 \n",
       "L 1825 4091 \n",
       "L 703 3866 \n",
       "L 703 4441 \n",
       "L 1819 4666 \n",
       "L 2450 4666 \n",
       "L 2450 531 \n",
       "L 3481 531 \n",
       "L 3481 0 \n",
       "L 794 0 \n",
       "L 794 531 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "        <path id=\"DejaVuSans-30\" d=\"M 2034 4250 \n",
       "Q 1547 4250 1301 3770 \n",
       "Q 1056 3291 1056 2328 \n",
       "Q 1056 1369 1301 889 \n",
       "Q 1547 409 2034 409 \n",
       "Q 2525 409 2770 889 \n",
       "Q 3016 1369 3016 2328 \n",
       "Q 3016 3291 2770 3770 \n",
       "Q 2525 4250 2034 4250 \n",
       "z\n",
       "M 2034 4750 \n",
       "Q 2819 4750 3233 4129 \n",
       "Q 3647 3509 3647 2328 \n",
       "Q 3647 1150 3233 529 \n",
       "Q 2819 -91 2034 -91 \n",
       "Q 1250 -91 836 529 \n",
       "Q 422 1150 422 2328 \n",
       "Q 422 3509 836 4129 \n",
       "Q 1250 4750 2034 4750 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       </defs>\n",
       "       <use xlink:href=\"#DejaVuSans-2212\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-31\" x=\"83.789062\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"147.412109\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_2\">\n",
       "     <g id=\"line2d_3\">\n",
       "      <path d=\"M 100.229261 143.1 \n",
       "L 100.229261 7.2 \n",
       "\" clip-path=\"url(#p62bcb874e4)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_4\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#mcbc85dced4\" x=\"100.229261\" y=\"143.1\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_2\">\n",
       "      <!-- −5 -->\n",
       "      <g transform=\"translate(92.858168 157.698438)scale(0.1 -0.1)\">\n",
       "       <defs>\n",
       "        <path id=\"DejaVuSans-35\" d=\"M 691 4666 \n",
       "L 3169 4666 \n",
       "L 3169 4134 \n",
       "L 1269 4134 \n",
       "L 1269 2991 \n",
       "Q 1406 3038 1543 3061 \n",
       "Q 1681 3084 1819 3084 \n",
       "Q 2600 3084 3056 2656 \n",
       "Q 3513 2228 3513 1497 \n",
       "Q 3513 744 3044 326 \n",
       "Q 2575 -91 1722 -91 \n",
       "Q 1428 -91 1123 -41 \n",
       "Q 819 9 494 109 \n",
       "L 494 744 \n",
       "Q 775 591 1075 516 \n",
       "Q 1375 441 1709 441 \n",
       "Q 2250 441 2565 725 \n",
       "Q 2881 1009 2881 1497 \n",
       "Q 2881 1984 2565 2268 \n",
       "Q 2250 2553 1709 2553 \n",
       "Q 1456 2553 1204 2497 \n",
       "Q 953 2441 691 2322 \n",
       "L 691 4666 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       </defs>\n",
       "       <use xlink:href=\"#DejaVuSans-2212\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-35\" x=\"83.789062\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_3\">\n",
       "     <g id=\"line2d_5\">\n",
       "      <path d=\"M 144.615625 143.1 \n",
       "L 144.615625 7.2 \n",
       "\" clip-path=\"url(#p62bcb874e4)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_6\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#mcbc85dced4\" x=\"144.615625\" y=\"143.1\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_3\">\n",
       "      <!-- 0 -->\n",
       "      <g transform=\"translate(141.434375 157.698438)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#DejaVuSans-30\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_4\">\n",
       "     <g id=\"line2d_7\">\n",
       "      <path d=\"M 189.001989 143.1 \n",
       "L 189.001989 7.2 \n",
       "\" clip-path=\"url(#p62bcb874e4)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_8\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#mcbc85dced4\" x=\"189.001989\" y=\"143.1\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_4\">\n",
       "      <!-- 5 -->\n",
       "      <g transform=\"translate(185.820739 157.698438)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#DejaVuSans-35\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_5\">\n",
       "     <g id=\"line2d_9\">\n",
       "      <path d=\"M 233.388352 143.1 \n",
       "L 233.388352 7.2 \n",
       "\" clip-path=\"url(#p62bcb874e4)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_10\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#mcbc85dced4\" x=\"233.388352\" y=\"143.1\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_5\">\n",
       "      <!-- 10 -->\n",
       "      <g transform=\"translate(227.025852 157.698438)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#DejaVuSans-31\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"63.623047\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"text_6\">\n",
       "     <!-- x -->\n",
       "     <g transform=\"translate(141.65625 171.376563)scale(0.1 -0.1)\">\n",
       "      <defs>\n",
       "       <path id=\"DejaVuSans-78\" d=\"M 3513 3500 \n",
       "L 2247 1797 \n",
       "L 3578 0 \n",
       "L 2900 0 \n",
       "L 1881 1375 \n",
       "L 863 0 \n",
       "L 184 0 \n",
       "L 1544 1831 \n",
       "L 300 3500 \n",
       "L 978 3500 \n",
       "L 1906 2253 \n",
       "L 2834 3500 \n",
       "L 3513 3500 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "      </defs>\n",
       "      <use xlink:href=\"#DejaVuSans-78\"/>\n",
       "     </g>\n",
       "    </g>\n",
       "   </g>\n",
       "   <g id=\"matplotlib.axis_2\">\n",
       "    <g id=\"ytick_1\">\n",
       "     <g id=\"line2d_11\">\n",
       "      <path d=\"M 46.965625 136.922727 \n",
       "L 242.265625 136.922727 \n",
       "\" clip-path=\"url(#p62bcb874e4)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_12\">\n",
       "      <defs>\n",
       "       <path id=\"m313674a0ef\" d=\"M 0 0 \n",
       "L -3.5 0 \n",
       "\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </defs>\n",
       "      <g>\n",
       "       <use xlink:href=\"#m313674a0ef\" x=\"46.965625\" y=\"136.922727\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_7\">\n",
       "      <!-- 0 -->\n",
       "      <g transform=\"translate(33.603125 140.721946)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#DejaVuSans-30\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_2\">\n",
       "     <g id=\"line2d_13\">\n",
       "      <path d=\"M 46.965625 112.213636 \n",
       "L 242.265625 112.213636 \n",
       "\" clip-path=\"url(#p62bcb874e4)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_14\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#m313674a0ef\" x=\"46.965625\" y=\"112.213636\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_8\">\n",
       "      <!-- 20 -->\n",
       "      <g transform=\"translate(27.240625 116.012855)scale(0.1 -0.1)\">\n",
       "       <defs>\n",
       "        <path id=\"DejaVuSans-32\" d=\"M 1228 531 \n",
       "L 3431 531 \n",
       "L 3431 0 \n",
       "L 469 0 \n",
       "L 469 531 \n",
       "Q 828 903 1448 1529 \n",
       "Q 2069 2156 2228 2338 \n",
       "Q 2531 2678 2651 2914 \n",
       "Q 2772 3150 2772 3378 \n",
       "Q 2772 3750 2511 3984 \n",
       "Q 2250 4219 1831 4219 \n",
       "Q 1534 4219 1204 4116 \n",
       "Q 875 4013 500 3803 \n",
       "L 500 4441 \n",
       "Q 881 4594 1212 4672 \n",
       "Q 1544 4750 1819 4750 \n",
       "Q 2544 4750 2975 4387 \n",
       "Q 3406 4025 3406 3419 \n",
       "Q 3406 3131 3298 2873 \n",
       "Q 3191 2616 2906 2266 \n",
       "Q 2828 2175 2409 1742 \n",
       "Q 1991 1309 1228 531 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       </defs>\n",
       "       <use xlink:href=\"#DejaVuSans-32\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"63.623047\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_3\">\n",
       "     <g id=\"line2d_15\">\n",
       "      <path d=\"M 46.965625 87.504545 \n",
       "L 242.265625 87.504545 \n",
       "\" clip-path=\"url(#p62bcb874e4)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_16\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#m313674a0ef\" x=\"46.965625\" y=\"87.504545\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_9\">\n",
       "      <!-- 40 -->\n",
       "      <g transform=\"translate(27.240625 91.303764)scale(0.1 -0.1)\">\n",
       "       <defs>\n",
       "        <path id=\"DejaVuSans-34\" d=\"M 2419 4116 \n",
       "L 825 1625 \n",
       "L 2419 1625 \n",
       "L 2419 4116 \n",
       "z\n",
       "M 2253 4666 \n",
       "L 3047 4666 \n",
       "L 3047 1625 \n",
       "L 3713 1625 \n",
       "L 3713 1100 \n",
       "L 3047 1100 \n",
       "L 3047 0 \n",
       "L 2419 0 \n",
       "L 2419 1100 \n",
       "L 313 1100 \n",
       "L 313 1709 \n",
       "L 2253 4666 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       </defs>\n",
       "       <use xlink:href=\"#DejaVuSans-34\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"63.623047\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_4\">\n",
       "     <g id=\"line2d_17\">\n",
       "      <path d=\"M 46.965625 62.795455 \n",
       "L 242.265625 62.795455 \n",
       "\" clip-path=\"url(#p62bcb874e4)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_18\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#m313674a0ef\" x=\"46.965625\" y=\"62.795455\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_10\">\n",
       "      <!-- 60 -->\n",
       "      <g transform=\"translate(27.240625 66.594673)scale(0.1 -0.1)\">\n",
       "       <defs>\n",
       "        <path id=\"DejaVuSans-36\" d=\"M 2113 2584 \n",
       "Q 1688 2584 1439 2293 \n",
       "Q 1191 2003 1191 1497 \n",
       "Q 1191 994 1439 701 \n",
       "Q 1688 409 2113 409 \n",
       "Q 2538 409 2786 701 \n",
       "Q 3034 994 3034 1497 \n",
       "Q 3034 2003 2786 2293 \n",
       "Q 2538 2584 2113 2584 \n",
       "z\n",
       "M 3366 4563 \n",
       "L 3366 3988 \n",
       "Q 3128 4100 2886 4159 \n",
       "Q 2644 4219 2406 4219 \n",
       "Q 1781 4219 1451 3797 \n",
       "Q 1122 3375 1075 2522 \n",
       "Q 1259 2794 1537 2939 \n",
       "Q 1816 3084 2150 3084 \n",
       "Q 2853 3084 3261 2657 \n",
       "Q 3669 2231 3669 1497 \n",
       "Q 3669 778 3244 343 \n",
       "Q 2819 -91 2113 -91 \n",
       "Q 1303 -91 875 529 \n",
       "Q 447 1150 447 2328 \n",
       "Q 447 3434 972 4092 \n",
       "Q 1497 4750 2381 4750 \n",
       "Q 2619 4750 2861 4703 \n",
       "Q 3103 4656 3366 4563 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       </defs>\n",
       "       <use xlink:href=\"#DejaVuSans-36\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"63.623047\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_5\">\n",
       "     <g id=\"line2d_19\">\n",
       "      <path d=\"M 46.965625 38.086364 \n",
       "L 242.265625 38.086364 \n",
       "\" clip-path=\"url(#p62bcb874e4)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_20\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#m313674a0ef\" x=\"46.965625\" y=\"38.086364\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_11\">\n",
       "      <!-- 80 -->\n",
       "      <g transform=\"translate(27.240625 41.885582)scale(0.1 -0.1)\">\n",
       "       <defs>\n",
       "        <path id=\"DejaVuSans-38\" d=\"M 2034 2216 \n",
       "Q 1584 2216 1326 1975 \n",
       "Q 1069 1734 1069 1313 \n",
       "Q 1069 891 1326 650 \n",
       "Q 1584 409 2034 409 \n",
       "Q 2484 409 2743 651 \n",
       "Q 3003 894 3003 1313 \n",
       "Q 3003 1734 2745 1975 \n",
       "Q 2488 2216 2034 2216 \n",
       "z\n",
       "M 1403 2484 \n",
       "Q 997 2584 770 2862 \n",
       "Q 544 3141 544 3541 \n",
       "Q 544 4100 942 4425 \n",
       "Q 1341 4750 2034 4750 \n",
       "Q 2731 4750 3128 4425 \n",
       "Q 3525 4100 3525 3541 \n",
       "Q 3525 3141 3298 2862 \n",
       "Q 3072 2584 2669 2484 \n",
       "Q 3125 2378 3379 2068 \n",
       "Q 3634 1759 3634 1313 \n",
       "Q 3634 634 3220 271 \n",
       "Q 2806 -91 2034 -91 \n",
       "Q 1263 -91 848 271 \n",
       "Q 434 634 434 1313 \n",
       "Q 434 1759 690 2068 \n",
       "Q 947 2378 1403 2484 \n",
       "z\n",
       "M 1172 3481 \n",
       "Q 1172 3119 1398 2916 \n",
       "Q 1625 2713 2034 2713 \n",
       "Q 2441 2713 2670 2916 \n",
       "Q 2900 3119 2900 3481 \n",
       "Q 2900 3844 2670 4047 \n",
       "Q 2441 4250 2034 4250 \n",
       "Q 1625 4250 1398 4047 \n",
       "Q 1172 3844 1172 3481 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       </defs>\n",
       "       <use xlink:href=\"#DejaVuSans-38\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"63.623047\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_6\">\n",
       "     <g id=\"line2d_21\">\n",
       "      <path d=\"M 46.965625 13.377273 \n",
       "L 242.265625 13.377273 \n",
       "\" clip-path=\"url(#p62bcb874e4)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_22\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#m313674a0ef\" x=\"46.965625\" y=\"13.377273\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_12\">\n",
       "      <!-- 100 -->\n",
       "      <g transform=\"translate(20.878125 17.176491)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#DejaVuSans-31\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"63.623047\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"127.246094\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"text_13\">\n",
       "     <!-- f(x) -->\n",
       "     <g transform=\"translate(14.798438 83.771094)rotate(-90)scale(0.1 -0.1)\">\n",
       "      <defs>\n",
       "       <path id=\"DejaVuSans-66\" d=\"M 2375 4863 \n",
       "L 2375 4384 \n",
       "L 1825 4384 \n",
       "Q 1516 4384 1395 4259 \n",
       "Q 1275 4134 1275 3809 \n",
       "L 1275 3500 \n",
       "L 2222 3500 \n",
       "L 2222 3053 \n",
       "L 1275 3053 \n",
       "L 1275 0 \n",
       "L 697 0 \n",
       "L 697 3053 \n",
       "L 147 3053 \n",
       "L 147 3500 \n",
       "L 697 3500 \n",
       "L 697 3744 \n",
       "Q 697 4328 969 4595 \n",
       "Q 1241 4863 1831 4863 \n",
       "L 2375 4863 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-28\" d=\"M 1984 4856 \n",
       "Q 1566 4138 1362 3434 \n",
       "Q 1159 2731 1159 2009 \n",
       "Q 1159 1288 1364 580 \n",
       "Q 1569 -128 1984 -844 \n",
       "L 1484 -844 \n",
       "Q 1016 -109 783 600 \n",
       "Q 550 1309 550 2009 \n",
       "Q 550 2706 781 3412 \n",
       "Q 1013 4119 1484 4856 \n",
       "L 1984 4856 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-29\" d=\"M 513 4856 \n",
       "L 1013 4856 \n",
       "Q 1481 4119 1714 3412 \n",
       "Q 1947 2706 1947 2009 \n",
       "Q 1947 1309 1714 600 \n",
       "Q 1481 -109 1013 -844 \n",
       "L 513 -844 \n",
       "Q 928 -128 1133 580 \n",
       "Q 1338 1288 1338 2009 \n",
       "Q 1338 2731 1133 3434 \n",
       "Q 928 4138 513 4856 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "      </defs>\n",
       "      <use xlink:href=\"#DejaVuSans-66\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-28\" x=\"35.205078\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-78\" x=\"74.21875\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-29\" x=\"133.398438\"/>\n",
       "     </g>\n",
       "    </g>\n",
       "   </g>\n",
       "   <g id=\"line2d_23\">\n",
       "    <path d=\"M 55.842898 13.377273 \n",
       "L 60.459075 25.891924 \n",
       "L 64.89772 37.295193 \n",
       "L 69.247573 47.871029 \n",
       "L 73.508668 57.655837 \n",
       "L 77.680989 66.685159 \n",
       "L 81.764535 74.993839 \n",
       "L 85.75931 82.615983 \n",
       "L 89.576536 89.431856 \n",
       "L 93.304987 95.648166 \n",
       "L 96.944671 101.29605 \n",
       "L 100.495581 106.405892 \n",
       "L 103.957717 111.00734 \n",
       "L 107.331081 115.129312 \n",
       "L 110.615671 118.799969 \n",
       "L 113.811488 122.046742 \n",
       "L 116.918535 124.89632 \n",
       "L 120.02558 127.443208 \n",
       "L 123.043852 129.627491 \n",
       "L 125.973351 131.474372 \n",
       "L 128.814079 133.008313 \n",
       "L 131.654806 134.289232 \n",
       "L 134.406761 135.288838 \n",
       "L 137.158716 136.050991 \n",
       "L 139.821898 136.562469 \n",
       "L 142.48508 136.851565 \n",
       "L 145.148261 136.91828 \n",
       "L 147.811443 136.762612 \n",
       "L 150.474625 136.384563 \n",
       "L 153.137807 135.784132 \n",
       "L 155.889762 134.930062 \n",
       "L 158.641715 133.838539 \n",
       "L 161.39367 132.50956 \n",
       "L 164.234398 130.888644 \n",
       "L 167.163897 128.952069 \n",
       "L 170.093397 126.746412 \n",
       "L 173.111671 124.19248 \n",
       "L 176.218715 121.265071 \n",
       "L 179.414535 117.938238 \n",
       "L 182.61035 114.29118 \n",
       "L 185.894944 110.20911 \n",
       "L 189.268309 105.664614 \n",
       "L 192.730444 100.62952 \n",
       "L 196.281354 95.074914 \n",
       "L 199.921034 88.971153 \n",
       "L 203.649489 82.287839 \n",
       "L 207.55549 74.818774 \n",
       "L 211.550261 66.685159 \n",
       "L 215.633807 57.853636 \n",
       "L 219.806119 48.290118 \n",
       "L 224.067214 37.959738 \n",
       "L 228.417076 26.826932 \n",
       "L 232.855721 14.855356 \n",
       "L 233.299578 13.624247 \n",
       "L 233.299578 13.624247 \n",
       "\" clip-path=\"url(#p62bcb874e4)\" style=\"fill: none; stroke: #1f77b4; stroke-width: 1.5; stroke-linecap: square\"/>\n",
       "   </g>\n",
       "   <g id=\"line2d_24\">\n",
       "    <path d=\"M 233.388352 13.377273 \n",
       "L 197.879261 92.446364 \n",
       "L 176.573807 120.911236 \n",
       "L 163.790534 131.158591 \n",
       "L 156.12057 134.847638 \n",
       "L 151.518592 136.175695 \n",
       "L 148.757405 136.653796 \n",
       "L 147.100693 136.825912 \n",
       "L 146.106666 136.887874 \n",
       "L 145.51025 136.91018 \n",
       "L 145.1524 136.91821 \n",
       "\" clip-path=\"url(#p62bcb874e4)\" style=\"fill: none; stroke: #ff7f0e; stroke-width: 1.5; stroke-linecap: square\"/>\n",
       "    <defs>\n",
       "     <path id=\"m1e30e819a8\" d=\"M 0 3 \n",
       "C 0.795609 3 1.55874 2.683901 2.12132 2.12132 \n",
       "C 2.683901 1.55874 3 0.795609 3 0 \n",
       "C 3 -0.795609 2.683901 -1.55874 2.12132 -2.12132 \n",
       "C 1.55874 -2.683901 0.795609 -3 0 -3 \n",
       "C -0.795609 -3 -1.55874 -2.683901 -2.12132 -2.12132 \n",
       "C -2.683901 -1.55874 -3 -0.795609 -3 0 \n",
       "C -3 0.795609 -2.683901 1.55874 -2.12132 2.12132 \n",
       "C -1.55874 2.683901 -0.795609 3 0 3 \n",
       "z\n",
       "\" style=\"stroke: #ff7f0e\"/>\n",
       "    </defs>\n",
       "    <g clip-path=\"url(#p62bcb874e4)\">\n",
       "     <use xlink:href=\"#m1e30e819a8\" x=\"233.388352\" y=\"13.377273\" style=\"fill: #ff7f0e; stroke: #ff7f0e\"/>\n",
       "     <use xlink:href=\"#m1e30e819a8\" x=\"197.879261\" y=\"92.446364\" style=\"fill: #ff7f0e; stroke: #ff7f0e\"/>\n",
       "     <use xlink:href=\"#m1e30e819a8\" x=\"176.573807\" y=\"120.911236\" style=\"fill: #ff7f0e; stroke: #ff7f0e\"/>\n",
       "     <use xlink:href=\"#m1e30e819a8\" x=\"163.790534\" y=\"131.158591\" style=\"fill: #ff7f0e; stroke: #ff7f0e\"/>\n",
       "     <use xlink:href=\"#m1e30e819a8\" x=\"156.12057\" y=\"134.847638\" style=\"fill: #ff7f0e; stroke: #ff7f0e\"/>\n",
       "     <use xlink:href=\"#m1e30e819a8\" x=\"151.518592\" y=\"136.175695\" style=\"fill: #ff7f0e; stroke: #ff7f0e\"/>\n",
       "     <use xlink:href=\"#m1e30e819a8\" x=\"148.757405\" y=\"136.653796\" style=\"fill: #ff7f0e; stroke: #ff7f0e\"/>\n",
       "     <use xlink:href=\"#m1e30e819a8\" x=\"147.100693\" y=\"136.825912\" style=\"fill: #ff7f0e; stroke: #ff7f0e\"/>\n",
       "     <use xlink:href=\"#m1e30e819a8\" x=\"146.106666\" y=\"136.887874\" style=\"fill: #ff7f0e; stroke: #ff7f0e\"/>\n",
       "     <use xlink:href=\"#m1e30e819a8\" x=\"145.51025\" y=\"136.91018\" style=\"fill: #ff7f0e; stroke: #ff7f0e\"/>\n",
       "     <use xlink:href=\"#m1e30e819a8\" x=\"145.1524\" y=\"136.91821\" style=\"fill: #ff7f0e; stroke: #ff7f0e\"/>\n",
       "    </g>\n",
       "   </g>\n",
       "   <g id=\"patch_3\">\n",
       "    <path d=\"M 46.965625 143.1 \n",
       "L 46.965625 7.2 \n",
       "\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_4\">\n",
       "    <path d=\"M 242.265625 143.1 \n",
       "L 242.265625 7.2 \n",
       "\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_5\">\n",
       "    <path d=\"M 46.965625 143.1 \n",
       "L 242.265625 143.1 \n",
       "\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_6\">\n",
       "    <path d=\"M 46.965625 7.2 \n",
       "L 242.265625 7.2 \n",
       "\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n",
       "   </g>\n",
       "  </g>\n",
       " </g>\n",
       " <defs>\n",
       "  <clipPath id=\"p62bcb874e4\">\n",
       "   <rect x=\"46.965625\" y=\"7.2\" width=\"195.3\" height=\"135.9\"/>\n",
       "  </clipPath>\n",
       " </defs>\n",
       "</svg>\n"
      ],
      "text/plain": [
       "<Figure size 252x180 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "def show_trace(results, f):\n",
    "    n = max(abs(min(results)), abs(max(results)))\n",
    "    f_line = torch.arange(-n, n, 0.01)\n",
    "    d2l.set_figsize()\n",
    "    d2l.plot([f_line, results], [[f(x) for x in f_line], [\n",
    "        f(x) for x in results]], 'x', 'f(x)', fmts=['-', '-o'])\n",
    "\n",
    "show_trace(results, f)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "467465ec",
   "metadata": {
    "origin_pos": 12
   },
   "source": [
    "### 学习率\n",
    ":label:`subsec_gd-learningrate`\n",
    "\n",
    "*学习率*（learning rate）决定目标函数能否收敛到局部最小值，以及何时收敛到最小值。\n",
    "学习率$\\eta$可由算法设计者设置。\n",
    "请注意，如果我们使用的学习率太小，将导致$x$的更新非常缓慢，需要更多的迭代。\n",
    "例如，考虑同一优化问题中$\\eta = 0.05$的进度。\n",
    "如下所示，尽管经过了10个步骤，我们仍然离最优解很远。\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "f785cac5",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-08-18T07:01:26.486541Z",
     "iopub.status.busy": "2023-08-18T07:01:26.485458Z",
     "iopub.status.idle": "2023-08-18T07:01:26.770061Z",
     "shell.execute_reply": "2023-08-18T07:01:26.768785Z"
    },
    "origin_pos": 13,
    "tab": [
     "pytorch"
    ]
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "epoch 10, x: 3.486784\n"
     ]
    },
    {
     "data": {
      "image/svg+xml": [
       "<?xml version=\"1.0\" encoding=\"utf-8\" standalone=\"no\"?>\n",
       "<!DOCTYPE svg PUBLIC \"-//W3C//DTD SVG 1.1//EN\"\n",
       "  \"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd\">\n",
       "<svg xmlns:xlink=\"http://www.w3.org/1999/xlink\" width=\"249.465625pt\" height=\"180.65625pt\" viewBox=\"0 0 249.465625 180.65625\" xmlns=\"http://www.w3.org/2000/svg\" version=\"1.1\">\n",
       " <metadata>\n",
       "  <rdf:RDF xmlns:dc=\"http://purl.org/dc/elements/1.1/\" xmlns:cc=\"http://creativecommons.org/ns#\" xmlns:rdf=\"http://www.w3.org/1999/02/22-rdf-syntax-ns#\">\n",
       "   <cc:Work>\n",
       "    <dc:type rdf:resource=\"http://purl.org/dc/dcmitype/StillImage\"/>\n",
       "    <dc:date>2023-08-18T07:01:26.717468</dc:date>\n",
       "    <dc:format>image/svg+xml</dc:format>\n",
       "    <dc:creator>\n",
       "     <cc:Agent>\n",
       "      <dc:title>Matplotlib v3.5.1, https://matplotlib.org/</dc:title>\n",
       "     </cc:Agent>\n",
       "    </dc:creator>\n",
       "   </cc:Work>\n",
       "  </rdf:RDF>\n",
       " </metadata>\n",
       " <defs>\n",
       "  <style type=\"text/css\">*{stroke-linejoin: round; stroke-linecap: butt}</style>\n",
       " </defs>\n",
       " <g id=\"figure_1\">\n",
       "  <g id=\"patch_1\">\n",
       "   <path d=\"M 0 180.65625 \n",
       "L 249.465625 180.65625 \n",
       "L 249.465625 0 \n",
       "L 0 0 \n",
       "L 0 180.65625 \n",
       "z\n",
       "\" style=\"fill: none\"/>\n",
       "  </g>\n",
       "  <g id=\"axes_1\">\n",
       "   <g id=\"patch_2\">\n",
       "    <path d=\"M 46.965625 143.1 \n",
       "L 242.265625 143.1 \n",
       "L 242.265625 7.2 \n",
       "L 46.965625 7.2 \n",
       "z\n",
       "\" style=\"fill: #ffffff\"/>\n",
       "   </g>\n",
       "   <g id=\"matplotlib.axis_1\">\n",
       "    <g id=\"xtick_1\">\n",
       "     <g id=\"line2d_1\">\n",
       "      <path d=\"M 55.842898 143.1 \n",
       "L 55.842898 7.2 \n",
       "\" clip-path=\"url(#p9c16b0ba75)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_2\">\n",
       "      <defs>\n",
       "       <path id=\"m55e3e10dda\" d=\"M 0 0 \n",
       "L 0 3.5 \n",
       "\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </defs>\n",
       "      <g>\n",
       "       <use xlink:href=\"#m55e3e10dda\" x=\"55.842898\" y=\"143.1\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_1\">\n",
       "      <!-- −10 -->\n",
       "      <g transform=\"translate(45.290554 157.698438)scale(0.1 -0.1)\">\n",
       "       <defs>\n",
       "        <path id=\"DejaVuSans-2212\" d=\"M 678 2272 \n",
       "L 4684 2272 \n",
       "L 4684 1741 \n",
       "L 678 1741 \n",
       "L 678 2272 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "        <path id=\"DejaVuSans-31\" d=\"M 794 531 \n",
       "L 1825 531 \n",
       "L 1825 4091 \n",
       "L 703 3866 \n",
       "L 703 4441 \n",
       "L 1819 4666 \n",
       "L 2450 4666 \n",
       "L 2450 531 \n",
       "L 3481 531 \n",
       "L 3481 0 \n",
       "L 794 0 \n",
       "L 794 531 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "        <path id=\"DejaVuSans-30\" d=\"M 2034 4250 \n",
       "Q 1547 4250 1301 3770 \n",
       "Q 1056 3291 1056 2328 \n",
       "Q 1056 1369 1301 889 \n",
       "Q 1547 409 2034 409 \n",
       "Q 2525 409 2770 889 \n",
       "Q 3016 1369 3016 2328 \n",
       "Q 3016 3291 2770 3770 \n",
       "Q 2525 4250 2034 4250 \n",
       "z\n",
       "M 2034 4750 \n",
       "Q 2819 4750 3233 4129 \n",
       "Q 3647 3509 3647 2328 \n",
       "Q 3647 1150 3233 529 \n",
       "Q 2819 -91 2034 -91 \n",
       "Q 1250 -91 836 529 \n",
       "Q 422 1150 422 2328 \n",
       "Q 422 3509 836 4129 \n",
       "Q 1250 4750 2034 4750 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       </defs>\n",
       "       <use xlink:href=\"#DejaVuSans-2212\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-31\" x=\"83.789062\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"147.412109\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_2\">\n",
       "     <g id=\"line2d_3\">\n",
       "      <path d=\"M 100.229261 143.1 \n",
       "L 100.229261 7.2 \n",
       "\" clip-path=\"url(#p9c16b0ba75)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_4\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#m55e3e10dda\" x=\"100.229261\" y=\"143.1\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_2\">\n",
       "      <!-- −5 -->\n",
       "      <g transform=\"translate(92.858168 157.698438)scale(0.1 -0.1)\">\n",
       "       <defs>\n",
       "        <path id=\"DejaVuSans-35\" d=\"M 691 4666 \n",
       "L 3169 4666 \n",
       "L 3169 4134 \n",
       "L 1269 4134 \n",
       "L 1269 2991 \n",
       "Q 1406 3038 1543 3061 \n",
       "Q 1681 3084 1819 3084 \n",
       "Q 2600 3084 3056 2656 \n",
       "Q 3513 2228 3513 1497 \n",
       "Q 3513 744 3044 326 \n",
       "Q 2575 -91 1722 -91 \n",
       "Q 1428 -91 1123 -41 \n",
       "Q 819 9 494 109 \n",
       "L 494 744 \n",
       "Q 775 591 1075 516 \n",
       "Q 1375 441 1709 441 \n",
       "Q 2250 441 2565 725 \n",
       "Q 2881 1009 2881 1497 \n",
       "Q 2881 1984 2565 2268 \n",
       "Q 2250 2553 1709 2553 \n",
       "Q 1456 2553 1204 2497 \n",
       "Q 953 2441 691 2322 \n",
       "L 691 4666 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       </defs>\n",
       "       <use xlink:href=\"#DejaVuSans-2212\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-35\" x=\"83.789062\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_3\">\n",
       "     <g id=\"line2d_5\">\n",
       "      <path d=\"M 144.615625 143.1 \n",
       "L 144.615625 7.2 \n",
       "\" clip-path=\"url(#p9c16b0ba75)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_6\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#m55e3e10dda\" x=\"144.615625\" y=\"143.1\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_3\">\n",
       "      <!-- 0 -->\n",
       "      <g transform=\"translate(141.434375 157.698438)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#DejaVuSans-30\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_4\">\n",
       "     <g id=\"line2d_7\">\n",
       "      <path d=\"M 189.001989 143.1 \n",
       "L 189.001989 7.2 \n",
       "\" clip-path=\"url(#p9c16b0ba75)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_8\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#m55e3e10dda\" x=\"189.001989\" y=\"143.1\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_4\">\n",
       "      <!-- 5 -->\n",
       "      <g transform=\"translate(185.820739 157.698438)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#DejaVuSans-35\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_5\">\n",
       "     <g id=\"line2d_9\">\n",
       "      <path d=\"M 233.388352 143.1 \n",
       "L 233.388352 7.2 \n",
       "\" clip-path=\"url(#p9c16b0ba75)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_10\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#m55e3e10dda\" x=\"233.388352\" y=\"143.1\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_5\">\n",
       "      <!-- 10 -->\n",
       "      <g transform=\"translate(227.025852 157.698438)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#DejaVuSans-31\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"63.623047\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"text_6\">\n",
       "     <!-- x -->\n",
       "     <g transform=\"translate(141.65625 171.376563)scale(0.1 -0.1)\">\n",
       "      <defs>\n",
       "       <path id=\"DejaVuSans-78\" d=\"M 3513 3500 \n",
       "L 2247 1797 \n",
       "L 3578 0 \n",
       "L 2900 0 \n",
       "L 1881 1375 \n",
       "L 863 0 \n",
       "L 184 0 \n",
       "L 1544 1831 \n",
       "L 300 3500 \n",
       "L 978 3500 \n",
       "L 1906 2253 \n",
       "L 2834 3500 \n",
       "L 3513 3500 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "      </defs>\n",
       "      <use xlink:href=\"#DejaVuSans-78\"/>\n",
       "     </g>\n",
       "    </g>\n",
       "   </g>\n",
       "   <g id=\"matplotlib.axis_2\">\n",
       "    <g id=\"ytick_1\">\n",
       "     <g id=\"line2d_11\">\n",
       "      <path d=\"M 46.965625 136.922727 \n",
       "L 242.265625 136.922727 \n",
       "\" clip-path=\"url(#p9c16b0ba75)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_12\">\n",
       "      <defs>\n",
       "       <path id=\"mf6d008a5b3\" d=\"M 0 0 \n",
       "L -3.5 0 \n",
       "\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </defs>\n",
       "      <g>\n",
       "       <use xlink:href=\"#mf6d008a5b3\" x=\"46.965625\" y=\"136.922727\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_7\">\n",
       "      <!-- 0 -->\n",
       "      <g transform=\"translate(33.603125 140.721946)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#DejaVuSans-30\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_2\">\n",
       "     <g id=\"line2d_13\">\n",
       "      <path d=\"M 46.965625 112.213636 \n",
       "L 242.265625 112.213636 \n",
       "\" clip-path=\"url(#p9c16b0ba75)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_14\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#mf6d008a5b3\" x=\"46.965625\" y=\"112.213636\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_8\">\n",
       "      <!-- 20 -->\n",
       "      <g transform=\"translate(27.240625 116.012855)scale(0.1 -0.1)\">\n",
       "       <defs>\n",
       "        <path id=\"DejaVuSans-32\" d=\"M 1228 531 \n",
       "L 3431 531 \n",
       "L 3431 0 \n",
       "L 469 0 \n",
       "L 469 531 \n",
       "Q 828 903 1448 1529 \n",
       "Q 2069 2156 2228 2338 \n",
       "Q 2531 2678 2651 2914 \n",
       "Q 2772 3150 2772 3378 \n",
       "Q 2772 3750 2511 3984 \n",
       "Q 2250 4219 1831 4219 \n",
       "Q 1534 4219 1204 4116 \n",
       "Q 875 4013 500 3803 \n",
       "L 500 4441 \n",
       "Q 881 4594 1212 4672 \n",
       "Q 1544 4750 1819 4750 \n",
       "Q 2544 4750 2975 4387 \n",
       "Q 3406 4025 3406 3419 \n",
       "Q 3406 3131 3298 2873 \n",
       "Q 3191 2616 2906 2266 \n",
       "Q 2828 2175 2409 1742 \n",
       "Q 1991 1309 1228 531 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       </defs>\n",
       "       <use xlink:href=\"#DejaVuSans-32\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"63.623047\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_3\">\n",
       "     <g id=\"line2d_15\">\n",
       "      <path d=\"M 46.965625 87.504545 \n",
       "L 242.265625 87.504545 \n",
       "\" clip-path=\"url(#p9c16b0ba75)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_16\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#mf6d008a5b3\" x=\"46.965625\" y=\"87.504545\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_9\">\n",
       "      <!-- 40 -->\n",
       "      <g transform=\"translate(27.240625 91.303764)scale(0.1 -0.1)\">\n",
       "       <defs>\n",
       "        <path id=\"DejaVuSans-34\" d=\"M 2419 4116 \n",
       "L 825 1625 \n",
       "L 2419 1625 \n",
       "L 2419 4116 \n",
       "z\n",
       "M 2253 4666 \n",
       "L 3047 4666 \n",
       "L 3047 1625 \n",
       "L 3713 1625 \n",
       "L 3713 1100 \n",
       "L 3047 1100 \n",
       "L 3047 0 \n",
       "L 2419 0 \n",
       "L 2419 1100 \n",
       "L 313 1100 \n",
       "L 313 1709 \n",
       "L 2253 4666 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       </defs>\n",
       "       <use xlink:href=\"#DejaVuSans-34\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"63.623047\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_4\">\n",
       "     <g id=\"line2d_17\">\n",
       "      <path d=\"M 46.965625 62.795455 \n",
       "L 242.265625 62.795455 \n",
       "\" clip-path=\"url(#p9c16b0ba75)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_18\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#mf6d008a5b3\" x=\"46.965625\" y=\"62.795455\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_10\">\n",
       "      <!-- 60 -->\n",
       "      <g transform=\"translate(27.240625 66.594673)scale(0.1 -0.1)\">\n",
       "       <defs>\n",
       "        <path id=\"DejaVuSans-36\" d=\"M 2113 2584 \n",
       "Q 1688 2584 1439 2293 \n",
       "Q 1191 2003 1191 1497 \n",
       "Q 1191 994 1439 701 \n",
       "Q 1688 409 2113 409 \n",
       "Q 2538 409 2786 701 \n",
       "Q 3034 994 3034 1497 \n",
       "Q 3034 2003 2786 2293 \n",
       "Q 2538 2584 2113 2584 \n",
       "z\n",
       "M 3366 4563 \n",
       "L 3366 3988 \n",
       "Q 3128 4100 2886 4159 \n",
       "Q 2644 4219 2406 4219 \n",
       "Q 1781 4219 1451 3797 \n",
       "Q 1122 3375 1075 2522 \n",
       "Q 1259 2794 1537 2939 \n",
       "Q 1816 3084 2150 3084 \n",
       "Q 2853 3084 3261 2657 \n",
       "Q 3669 2231 3669 1497 \n",
       "Q 3669 778 3244 343 \n",
       "Q 2819 -91 2113 -91 \n",
       "Q 1303 -91 875 529 \n",
       "Q 447 1150 447 2328 \n",
       "Q 447 3434 972 4092 \n",
       "Q 1497 4750 2381 4750 \n",
       "Q 2619 4750 2861 4703 \n",
       "Q 3103 4656 3366 4563 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       </defs>\n",
       "       <use xlink:href=\"#DejaVuSans-36\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"63.623047\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_5\">\n",
       "     <g id=\"line2d_19\">\n",
       "      <path d=\"M 46.965625 38.086364 \n",
       "L 242.265625 38.086364 \n",
       "\" clip-path=\"url(#p9c16b0ba75)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_20\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#mf6d008a5b3\" x=\"46.965625\" y=\"38.086364\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_11\">\n",
       "      <!-- 80 -->\n",
       "      <g transform=\"translate(27.240625 41.885582)scale(0.1 -0.1)\">\n",
       "       <defs>\n",
       "        <path id=\"DejaVuSans-38\" d=\"M 2034 2216 \n",
       "Q 1584 2216 1326 1975 \n",
       "Q 1069 1734 1069 1313 \n",
       "Q 1069 891 1326 650 \n",
       "Q 1584 409 2034 409 \n",
       "Q 2484 409 2743 651 \n",
       "Q 3003 894 3003 1313 \n",
       "Q 3003 1734 2745 1975 \n",
       "Q 2488 2216 2034 2216 \n",
       "z\n",
       "M 1403 2484 \n",
       "Q 997 2584 770 2862 \n",
       "Q 544 3141 544 3541 \n",
       "Q 544 4100 942 4425 \n",
       "Q 1341 4750 2034 4750 \n",
       "Q 2731 4750 3128 4425 \n",
       "Q 3525 4100 3525 3541 \n",
       "Q 3525 3141 3298 2862 \n",
       "Q 3072 2584 2669 2484 \n",
       "Q 3125 2378 3379 2068 \n",
       "Q 3634 1759 3634 1313 \n",
       "Q 3634 634 3220 271 \n",
       "Q 2806 -91 2034 -91 \n",
       "Q 1263 -91 848 271 \n",
       "Q 434 634 434 1313 \n",
       "Q 434 1759 690 2068 \n",
       "Q 947 2378 1403 2484 \n",
       "z\n",
       "M 1172 3481 \n",
       "Q 1172 3119 1398 2916 \n",
       "Q 1625 2713 2034 2713 \n",
       "Q 2441 2713 2670 2916 \n",
       "Q 2900 3119 2900 3481 \n",
       "Q 2900 3844 2670 4047 \n",
       "Q 2441 4250 2034 4250 \n",
       "Q 1625 4250 1398 4047 \n",
       "Q 1172 3844 1172 3481 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       </defs>\n",
       "       <use xlink:href=\"#DejaVuSans-38\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"63.623047\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_6\">\n",
       "     <g id=\"line2d_21\">\n",
       "      <path d=\"M 46.965625 13.377273 \n",
       "L 242.265625 13.377273 \n",
       "\" clip-path=\"url(#p9c16b0ba75)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_22\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#mf6d008a5b3\" x=\"46.965625\" y=\"13.377273\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_12\">\n",
       "      <!-- 100 -->\n",
       "      <g transform=\"translate(20.878125 17.176491)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#DejaVuSans-31\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"63.623047\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"127.246094\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"text_13\">\n",
       "     <!-- f(x) -->\n",
       "     <g transform=\"translate(14.798438 83.771094)rotate(-90)scale(0.1 -0.1)\">\n",
       "      <defs>\n",
       "       <path id=\"DejaVuSans-66\" d=\"M 2375 4863 \n",
       "L 2375 4384 \n",
       "L 1825 4384 \n",
       "Q 1516 4384 1395 4259 \n",
       "Q 1275 4134 1275 3809 \n",
       "L 1275 3500 \n",
       "L 2222 3500 \n",
       "L 2222 3053 \n",
       "L 1275 3053 \n",
       "L 1275 0 \n",
       "L 697 0 \n",
       "L 697 3053 \n",
       "L 147 3053 \n",
       "L 147 3500 \n",
       "L 697 3500 \n",
       "L 697 3744 \n",
       "Q 697 4328 969 4595 \n",
       "Q 1241 4863 1831 4863 \n",
       "L 2375 4863 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-28\" d=\"M 1984 4856 \n",
       "Q 1566 4138 1362 3434 \n",
       "Q 1159 2731 1159 2009 \n",
       "Q 1159 1288 1364 580 \n",
       "Q 1569 -128 1984 -844 \n",
       "L 1484 -844 \n",
       "Q 1016 -109 783 600 \n",
       "Q 550 1309 550 2009 \n",
       "Q 550 2706 781 3412 \n",
       "Q 1013 4119 1484 4856 \n",
       "L 1984 4856 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-29\" d=\"M 513 4856 \n",
       "L 1013 4856 \n",
       "Q 1481 4119 1714 3412 \n",
       "Q 1947 2706 1947 2009 \n",
       "Q 1947 1309 1714 600 \n",
       "Q 1481 -109 1013 -844 \n",
       "L 513 -844 \n",
       "Q 928 -128 1133 580 \n",
       "Q 1338 1288 1338 2009 \n",
       "Q 1338 2731 1133 3434 \n",
       "Q 928 4138 513 4856 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "      </defs>\n",
       "      <use xlink:href=\"#DejaVuSans-66\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-28\" x=\"35.205078\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-78\" x=\"74.21875\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-29\" x=\"133.398438\"/>\n",
       "     </g>\n",
       "    </g>\n",
       "   </g>\n",
       "   <g id=\"line2d_23\">\n",
       "    <path d=\"M 55.842898 13.377273 \n",
       "L 60.459075 25.891924 \n",
       "L 64.89772 37.295193 \n",
       "L 69.247573 47.871029 \n",
       "L 73.508668 57.655837 \n",
       "L 77.680989 66.685159 \n",
       "L 81.764535 74.993839 \n",
       "L 85.75931 82.615983 \n",
       "L 89.576536 89.431856 \n",
       "L 93.304987 95.648166 \n",
       "L 96.944671 101.29605 \n",
       "L 100.495581 106.405892 \n",
       "L 103.957717 111.00734 \n",
       "L 107.331081 115.129312 \n",
       "L 110.615671 118.799969 \n",
       "L 113.811488 122.046742 \n",
       "L 116.918535 124.89632 \n",
       "L 120.02558 127.443208 \n",
       "L 123.043852 129.627491 \n",
       "L 125.973351 131.474372 \n",
       "L 128.814079 133.008313 \n",
       "L 131.654806 134.289232 \n",
       "L 134.406761 135.288838 \n",
       "L 137.158716 136.050991 \n",
       "L 139.821898 136.562469 \n",
       "L 142.48508 136.851565 \n",
       "L 145.148261 136.91828 \n",
       "L 147.811443 136.762612 \n",
       "L 150.474625 136.384563 \n",
       "L 153.137807 135.784132 \n",
       "L 155.889762 134.930062 \n",
       "L 158.641715 133.838539 \n",
       "L 161.39367 132.50956 \n",
       "L 164.234398 130.888644 \n",
       "L 167.163897 128.952069 \n",
       "L 170.093397 126.746412 \n",
       "L 173.111671 124.19248 \n",
       "L 176.218715 121.265071 \n",
       "L 179.414535 117.938238 \n",
       "L 182.61035 114.29118 \n",
       "L 185.894944 110.20911 \n",
       "L 189.268309 105.664614 \n",
       "L 192.730444 100.62952 \n",
       "L 196.281354 95.074914 \n",
       "L 199.921034 88.971153 \n",
       "L 203.649489 82.287839 \n",
       "L 207.55549 74.818774 \n",
       "L 211.550261 66.685159 \n",
       "L 215.633807 57.853636 \n",
       "L 219.806119 48.290118 \n",
       "L 224.067214 37.959738 \n",
       "L 228.417076 26.826932 \n",
       "L 232.855721 14.855356 \n",
       "L 233.299578 13.624247 \n",
       "L 233.299578 13.624247 \n",
       "\" clip-path=\"url(#p9c16b0ba75)\" style=\"fill: none; stroke: #1f77b4; stroke-width: 1.5; stroke-linecap: square\"/>\n",
       "   </g>\n",
       "   <g id=\"line2d_24\">\n",
       "    <path d=\"M 233.388352 13.377273 \n",
       "L 224.51108 36.850909 \n",
       "L 216.521534 55.864555 \n",
       "L 209.330943 71.265607 \n",
       "L 202.859411 83.74046 \n",
       "L 197.035033 93.845091 \n",
       "L 191.793092 102.029842 \n",
       "L 187.075345 108.65949 \n",
       "L 182.829373 114.029505 \n",
       "L 179.007998 118.379217 \n",
       "L 175.568761 121.902484 \n",
       "\" clip-path=\"url(#p9c16b0ba75)\" style=\"fill: none; stroke: #ff7f0e; stroke-width: 1.5; stroke-linecap: square\"/>\n",
       "    <defs>\n",
       "     <path id=\"mbc97294271\" d=\"M 0 3 \n",
       "C 0.795609 3 1.55874 2.683901 2.12132 2.12132 \n",
       "C 2.683901 1.55874 3 0.795609 3 0 \n",
       "C 3 -0.795609 2.683901 -1.55874 2.12132 -2.12132 \n",
       "C 1.55874 -2.683901 0.795609 -3 0 -3 \n",
       "C -0.795609 -3 -1.55874 -2.683901 -2.12132 -2.12132 \n",
       "C -2.683901 -1.55874 -3 -0.795609 -3 0 \n",
       "C -3 0.795609 -2.683901 1.55874 -2.12132 2.12132 \n",
       "C -1.55874 2.683901 -0.795609 3 0 3 \n",
       "z\n",
       "\" style=\"stroke: #ff7f0e\"/>\n",
       "    </defs>\n",
       "    <g clip-path=\"url(#p9c16b0ba75)\">\n",
       "     <use xlink:href=\"#mbc97294271\" x=\"233.388352\" y=\"13.377273\" style=\"fill: #ff7f0e; stroke: #ff7f0e\"/>\n",
       "     <use xlink:href=\"#mbc97294271\" x=\"224.51108\" y=\"36.850909\" style=\"fill: #ff7f0e; stroke: #ff7f0e\"/>\n",
       "     <use xlink:href=\"#mbc97294271\" x=\"216.521534\" y=\"55.864555\" style=\"fill: #ff7f0e; stroke: #ff7f0e\"/>\n",
       "     <use xlink:href=\"#mbc97294271\" x=\"209.330943\" y=\"71.265607\" style=\"fill: #ff7f0e; stroke: #ff7f0e\"/>\n",
       "     <use xlink:href=\"#mbc97294271\" x=\"202.859411\" y=\"83.74046\" style=\"fill: #ff7f0e; stroke: #ff7f0e\"/>\n",
       "     <use xlink:href=\"#mbc97294271\" x=\"197.035033\" y=\"93.845091\" style=\"fill: #ff7f0e; stroke: #ff7f0e\"/>\n",
       "     <use xlink:href=\"#mbc97294271\" x=\"191.793092\" y=\"102.029842\" style=\"fill: #ff7f0e; stroke: #ff7f0e\"/>\n",
       "     <use xlink:href=\"#mbc97294271\" x=\"187.075345\" y=\"108.65949\" style=\"fill: #ff7f0e; stroke: #ff7f0e\"/>\n",
       "     <use xlink:href=\"#mbc97294271\" x=\"182.829373\" y=\"114.029505\" style=\"fill: #ff7f0e; stroke: #ff7f0e\"/>\n",
       "     <use xlink:href=\"#mbc97294271\" x=\"179.007998\" y=\"118.379217\" style=\"fill: #ff7f0e; stroke: #ff7f0e\"/>\n",
       "     <use xlink:href=\"#mbc97294271\" x=\"175.568761\" y=\"121.902484\" style=\"fill: #ff7f0e; stroke: #ff7f0e\"/>\n",
       "    </g>\n",
       "   </g>\n",
       "   <g id=\"patch_3\">\n",
       "    <path d=\"M 46.965625 143.1 \n",
       "L 46.965625 7.2 \n",
       "\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_4\">\n",
       "    <path d=\"M 242.265625 143.1 \n",
       "L 242.265625 7.2 \n",
       "\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_5\">\n",
       "    <path d=\"M 46.965625 143.1 \n",
       "L 242.265625 143.1 \n",
       "\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_6\">\n",
       "    <path d=\"M 46.965625 7.2 \n",
       "L 242.265625 7.2 \n",
       "\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n",
       "   </g>\n",
       "  </g>\n",
       " </g>\n",
       " <defs>\n",
       "  <clipPath id=\"p9c16b0ba75\">\n",
       "   <rect x=\"46.965625\" y=\"7.2\" width=\"195.3\" height=\"135.9\"/>\n",
       "  </clipPath>\n",
       " </defs>\n",
       "</svg>\n"
      ],
      "text/plain": [
       "<Figure size 252x180 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "show_trace(gd(0.05, f_grad), f)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ffd34ac9",
   "metadata": {
    "origin_pos": 14
   },
   "source": [
    "相反，如果我们使用过高的学习率，$\\left|\\eta f'(x)\\right|$对于一阶泰勒展开式可能太大。\n",
    "也就是说， :eqref:`gd-taylor`中的$\\mathcal{O}(\\eta^2 f'^2(x))$可能变得显著了。\n",
    "在这种情况下，$x$的迭代不能保证降低$f(x)$的值。\n",
    "例如，当学习率为$\\eta=1.1$时，$x$超出了最优解$x=0$并逐渐发散。\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "dabdbe10",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-08-18T07:01:26.775496Z",
     "iopub.status.busy": "2023-08-18T07:01:26.774773Z",
     "iopub.status.idle": "2023-08-18T07:01:27.351226Z",
     "shell.execute_reply": "2023-08-18T07:01:27.349971Z"
    },
    "origin_pos": 15,
    "tab": [
     "pytorch"
    ]
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "epoch 10, x: 61.917364\n"
     ]
    },
    {
     "data": {
      "image/svg+xml": [
       "<?xml version=\"1.0\" encoding=\"utf-8\" standalone=\"no\"?>\n",
       "<!DOCTYPE svg PUBLIC \"-//W3C//DTD SVG 1.1//EN\"\n",
       "  \"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd\">\n",
       "<svg xmlns:xlink=\"http://www.w3.org/1999/xlink\" width=\"255.828125pt\" height=\"183.635382pt\" viewBox=\"0 0 255.828125 183.635382\" xmlns=\"http://www.w3.org/2000/svg\" version=\"1.1\">\n",
       " <metadata>\n",
       "  <rdf:RDF xmlns:dc=\"http://purl.org/dc/elements/1.1/\" xmlns:cc=\"http://creativecommons.org/ns#\" xmlns:rdf=\"http://www.w3.org/1999/02/22-rdf-syntax-ns#\">\n",
       "   <cc:Work>\n",
       "    <dc:type rdf:resource=\"http://purl.org/dc/dcmitype/StillImage\"/>\n",
       "    <dc:date>2023-08-18T07:01:27.300248</dc:date>\n",
       "    <dc:format>image/svg+xml</dc:format>\n",
       "    <dc:creator>\n",
       "     <cc:Agent>\n",
       "      <dc:title>Matplotlib v3.5.1, https://matplotlib.org/</dc:title>\n",
       "     </cc:Agent>\n",
       "    </dc:creator>\n",
       "   </cc:Work>\n",
       "  </rdf:RDF>\n",
       " </metadata>\n",
       " <defs>\n",
       "  <style type=\"text/css\">*{stroke-linejoin: round; stroke-linecap: butt}</style>\n",
       " </defs>\n",
       " <g id=\"figure_1\">\n",
       "  <g id=\"patch_1\">\n",
       "   <path d=\"M 0 183.635382 \n",
       "L 255.828125 183.635382 \n",
       "L 255.828125 0 \n",
       "L 0 0 \n",
       "L 0 183.635382 \n",
       "z\n",
       "\" style=\"fill: none\"/>\n",
       "  </g>\n",
       "  <g id=\"axes_1\">\n",
       "   <g id=\"patch_2\">\n",
       "    <path d=\"M 53.328125 146.079132 \n",
       "L 248.628125 146.079132 \n",
       "L 248.628125 10.179132 \n",
       "L 53.328125 10.179132 \n",
       "z\n",
       "\" style=\"fill: #ffffff\"/>\n",
       "   </g>\n",
       "   <g id=\"matplotlib.axis_1\">\n",
       "    <g id=\"xtick_1\">\n",
       "     <g id=\"line2d_1\">\n",
       "      <path d=\"M 79.291672 146.079132 \n",
       "L 79.291672 10.179132 \n",
       "\" clip-path=\"url(#p5e3e8d099e)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_2\">\n",
       "      <defs>\n",
       "       <path id=\"mefc5dc4dba\" d=\"M 0 0 \n",
       "L 0 3.5 \n",
       "\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </defs>\n",
       "      <g>\n",
       "       <use xlink:href=\"#mefc5dc4dba\" x=\"79.291672\" y=\"146.079132\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_1\">\n",
       "      <!-- −50 -->\n",
       "      <g transform=\"translate(68.739328 160.677569)scale(0.1 -0.1)\">\n",
       "       <defs>\n",
       "        <path id=\"DejaVuSans-2212\" d=\"M 678 2272 \n",
       "L 4684 2272 \n",
       "L 4684 1741 \n",
       "L 678 1741 \n",
       "L 678 2272 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "        <path id=\"DejaVuSans-35\" d=\"M 691 4666 \n",
       "L 3169 4666 \n",
       "L 3169 4134 \n",
       "L 1269 4134 \n",
       "L 1269 2991 \n",
       "Q 1406 3038 1543 3061 \n",
       "Q 1681 3084 1819 3084 \n",
       "Q 2600 3084 3056 2656 \n",
       "Q 3513 2228 3513 1497 \n",
       "Q 3513 744 3044 326 \n",
       "Q 2575 -91 1722 -91 \n",
       "Q 1428 -91 1123 -41 \n",
       "Q 819 9 494 109 \n",
       "L 494 744 \n",
       "Q 775 591 1075 516 \n",
       "Q 1375 441 1709 441 \n",
       "Q 2250 441 2565 725 \n",
       "Q 2881 1009 2881 1497 \n",
       "Q 2881 1984 2565 2268 \n",
       "Q 2250 2553 1709 2553 \n",
       "Q 1456 2553 1204 2497 \n",
       "Q 953 2441 691 2322 \n",
       "L 691 4666 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "        <path id=\"DejaVuSans-30\" d=\"M 2034 4250 \n",
       "Q 1547 4250 1301 3770 \n",
       "Q 1056 3291 1056 2328 \n",
       "Q 1056 1369 1301 889 \n",
       "Q 1547 409 2034 409 \n",
       "Q 2525 409 2770 889 \n",
       "Q 3016 1369 3016 2328 \n",
       "Q 3016 3291 2770 3770 \n",
       "Q 2525 4250 2034 4250 \n",
       "z\n",
       "M 2034 4750 \n",
       "Q 2819 4750 3233 4129 \n",
       "Q 3647 3509 3647 2328 \n",
       "Q 3647 1150 3233 529 \n",
       "Q 2819 -91 2034 -91 \n",
       "Q 1250 -91 836 529 \n",
       "Q 422 1150 422 2328 \n",
       "Q 422 3509 836 4129 \n",
       "Q 1250 4750 2034 4750 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       </defs>\n",
       "       <use xlink:href=\"#DejaVuSans-2212\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-35\" x=\"83.789062\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"147.412109\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_2\">\n",
       "     <g id=\"line2d_3\">\n",
       "      <path d=\"M 115.134899 146.079132 \n",
       "L 115.134899 10.179132 \n",
       "\" clip-path=\"url(#p5e3e8d099e)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_4\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#mefc5dc4dba\" x=\"115.134899\" y=\"146.079132\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_2\">\n",
       "      <!-- −25 -->\n",
       "      <g transform=\"translate(104.582555 160.677569)scale(0.1 -0.1)\">\n",
       "       <defs>\n",
       "        <path id=\"DejaVuSans-32\" d=\"M 1228 531 \n",
       "L 3431 531 \n",
       "L 3431 0 \n",
       "L 469 0 \n",
       "L 469 531 \n",
       "Q 828 903 1448 1529 \n",
       "Q 2069 2156 2228 2338 \n",
       "Q 2531 2678 2651 2914 \n",
       "Q 2772 3150 2772 3378 \n",
       "Q 2772 3750 2511 3984 \n",
       "Q 2250 4219 1831 4219 \n",
       "Q 1534 4219 1204 4116 \n",
       "Q 875 4013 500 3803 \n",
       "L 500 4441 \n",
       "Q 881 4594 1212 4672 \n",
       "Q 1544 4750 1819 4750 \n",
       "Q 2544 4750 2975 4387 \n",
       "Q 3406 4025 3406 3419 \n",
       "Q 3406 3131 3298 2873 \n",
       "Q 3191 2616 2906 2266 \n",
       "Q 2828 2175 2409 1742 \n",
       "Q 1991 1309 1228 531 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       </defs>\n",
       "       <use xlink:href=\"#DejaVuSans-2212\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-32\" x=\"83.789062\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-35\" x=\"147.412109\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_3\">\n",
       "     <g id=\"line2d_5\">\n",
       "      <path d=\"M 150.978126 146.079132 \n",
       "L 150.978126 10.179132 \n",
       "\" clip-path=\"url(#p5e3e8d099e)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_6\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#mefc5dc4dba\" x=\"150.978126\" y=\"146.079132\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_3\">\n",
       "      <!-- 0 -->\n",
       "      <g transform=\"translate(147.796876 160.677569)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#DejaVuSans-30\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_4\">\n",
       "     <g id=\"line2d_7\">\n",
       "      <path d=\"M 186.821353 146.079132 \n",
       "L 186.821353 10.179132 \n",
       "\" clip-path=\"url(#p5e3e8d099e)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_8\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#mefc5dc4dba\" x=\"186.821353\" y=\"146.079132\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_4\">\n",
       "      <!-- 25 -->\n",
       "      <g transform=\"translate(180.458853 160.677569)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#DejaVuSans-32\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-35\" x=\"63.623047\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_5\">\n",
       "     <g id=\"line2d_9\">\n",
       "      <path d=\"M 222.664581 146.079132 \n",
       "L 222.664581 10.179132 \n",
       "\" clip-path=\"url(#p5e3e8d099e)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_10\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#mefc5dc4dba\" x=\"222.664581\" y=\"146.079132\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_5\">\n",
       "      <!-- 50 -->\n",
       "      <g transform=\"translate(216.302081 160.677569)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#DejaVuSans-35\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"63.623047\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"text_6\">\n",
       "     <!-- x -->\n",
       "     <g transform=\"translate(148.01875 174.355694)scale(0.1 -0.1)\">\n",
       "      <defs>\n",
       "       <path id=\"DejaVuSans-78\" d=\"M 3513 3500 \n",
       "L 2247 1797 \n",
       "L 3578 0 \n",
       "L 2900 0 \n",
       "L 1881 1375 \n",
       "L 863 0 \n",
       "L 184 0 \n",
       "L 1544 1831 \n",
       "L 300 3500 \n",
       "L 978 3500 \n",
       "L 1906 2253 \n",
       "L 2834 3500 \n",
       "L 3513 3500 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "      </defs>\n",
       "      <use xlink:href=\"#DejaVuSans-78\"/>\n",
       "     </g>\n",
       "    </g>\n",
       "   </g>\n",
       "   <g id=\"matplotlib.axis_2\">\n",
       "    <g id=\"ytick_1\">\n",
       "     <g id=\"line2d_11\">\n",
       "      <path d=\"M 53.328125 139.901859 \n",
       "L 248.628125 139.901859 \n",
       "\" clip-path=\"url(#p5e3e8d099e)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_12\">\n",
       "      <defs>\n",
       "       <path id=\"me500198b42\" d=\"M 0 0 \n",
       "L -3.5 0 \n",
       "\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </defs>\n",
       "      <g>\n",
       "       <use xlink:href=\"#me500198b42\" x=\"53.328125\" y=\"139.901859\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_7\">\n",
       "      <!-- 0 -->\n",
       "      <g transform=\"translate(39.965625 143.701078)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#DejaVuSans-30\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_2\">\n",
       "     <g id=\"line2d_13\">\n",
       "      <path d=\"M 53.328125 107.676199 \n",
       "L 248.628125 107.676199 \n",
       "\" clip-path=\"url(#p5e3e8d099e)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_14\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#me500198b42\" x=\"53.328125\" y=\"107.676199\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_8\">\n",
       "      <!-- 1000 -->\n",
       "      <g transform=\"translate(20.878125 111.475418)scale(0.1 -0.1)\">\n",
       "       <defs>\n",
       "        <path id=\"DejaVuSans-31\" d=\"M 794 531 \n",
       "L 1825 531 \n",
       "L 1825 4091 \n",
       "L 703 3866 \n",
       "L 703 4441 \n",
       "L 1819 4666 \n",
       "L 2450 4666 \n",
       "L 2450 531 \n",
       "L 3481 531 \n",
       "L 3481 0 \n",
       "L 794 0 \n",
       "L 794 531 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       </defs>\n",
       "       <use xlink:href=\"#DejaVuSans-31\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"63.623047\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"127.246094\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"190.869141\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_3\">\n",
       "     <g id=\"line2d_15\">\n",
       "      <path d=\"M 53.328125 75.450539 \n",
       "L 248.628125 75.450539 \n",
       "\" clip-path=\"url(#p5e3e8d099e)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_16\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#me500198b42\" x=\"53.328125\" y=\"75.450539\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_9\">\n",
       "      <!-- 2000 -->\n",
       "      <g transform=\"translate(20.878125 79.249758)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#DejaVuSans-32\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"63.623047\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"127.246094\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"190.869141\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_4\">\n",
       "     <g id=\"line2d_17\">\n",
       "      <path d=\"M 53.328125 43.224879 \n",
       "L 248.628125 43.224879 \n",
       "\" clip-path=\"url(#p5e3e8d099e)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_18\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#me500198b42\" x=\"53.328125\" y=\"43.224879\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_10\">\n",
       "      <!-- 3000 -->\n",
       "      <g transform=\"translate(20.878125 47.024098)scale(0.1 -0.1)\">\n",
       "       <defs>\n",
       "        <path id=\"DejaVuSans-33\" d=\"M 2597 2516 \n",
       "Q 3050 2419 3304 2112 \n",
       "Q 3559 1806 3559 1356 \n",
       "Q 3559 666 3084 287 \n",
       "Q 2609 -91 1734 -91 \n",
       "Q 1441 -91 1130 -33 \n",
       "Q 819 25 488 141 \n",
       "L 488 750 \n",
       "Q 750 597 1062 519 \n",
       "Q 1375 441 1716 441 \n",
       "Q 2309 441 2620 675 \n",
       "Q 2931 909 2931 1356 \n",
       "Q 2931 1769 2642 2001 \n",
       "Q 2353 2234 1838 2234 \n",
       "L 1294 2234 \n",
       "L 1294 2753 \n",
       "L 1863 2753 \n",
       "Q 2328 2753 2575 2939 \n",
       "Q 2822 3125 2822 3475 \n",
       "Q 2822 3834 2567 4026 \n",
       "Q 2313 4219 1838 4219 \n",
       "Q 1578 4219 1281 4162 \n",
       "Q 984 4106 628 3988 \n",
       "L 628 4550 \n",
       "Q 988 4650 1302 4700 \n",
       "Q 1616 4750 1894 4750 \n",
       "Q 2613 4750 3031 4423 \n",
       "Q 3450 4097 3450 3541 \n",
       "Q 3450 3153 3228 2886 \n",
       "Q 3006 2619 2597 2516 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       </defs>\n",
       "       <use xlink:href=\"#DejaVuSans-33\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"63.623047\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"127.246094\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"190.869141\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_5\">\n",
       "     <g id=\"line2d_19\">\n",
       "      <path d=\"M 53.328125 10.999219 \n",
       "L 248.628125 10.999219 \n",
       "\" clip-path=\"url(#p5e3e8d099e)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_20\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#me500198b42\" x=\"53.328125\" y=\"10.999219\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_11\">\n",
       "      <!-- 4000 -->\n",
       "      <g transform=\"translate(20.878125 14.798437)scale(0.1 -0.1)\">\n",
       "       <defs>\n",
       "        <path id=\"DejaVuSans-34\" d=\"M 2419 4116 \n",
       "L 825 1625 \n",
       "L 2419 1625 \n",
       "L 2419 4116 \n",
       "z\n",
       "M 2253 4666 \n",
       "L 3047 4666 \n",
       "L 3047 1625 \n",
       "L 3713 1625 \n",
       "L 3713 1100 \n",
       "L 3047 1100 \n",
       "L 3047 0 \n",
       "L 2419 0 \n",
       "L 2419 1100 \n",
       "L 313 1100 \n",
       "L 313 1709 \n",
       "L 2253 4666 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       </defs>\n",
       "       <use xlink:href=\"#DejaVuSans-34\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"63.623047\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"127.246094\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"190.869141\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"text_12\">\n",
       "     <!-- f(x) -->\n",
       "     <g transform=\"translate(14.798437 86.750225)rotate(-90)scale(0.1 -0.1)\">\n",
       "      <defs>\n",
       "       <path id=\"DejaVuSans-66\" d=\"M 2375 4863 \n",
       "L 2375 4384 \n",
       "L 1825 4384 \n",
       "Q 1516 4384 1395 4259 \n",
       "Q 1275 4134 1275 3809 \n",
       "L 1275 3500 \n",
       "L 2222 3500 \n",
       "L 2222 3053 \n",
       "L 1275 3053 \n",
       "L 1275 0 \n",
       "L 697 0 \n",
       "L 697 3053 \n",
       "L 147 3053 \n",
       "L 147 3500 \n",
       "L 697 3500 \n",
       "L 697 3744 \n",
       "Q 697 4328 969 4595 \n",
       "Q 1241 4863 1831 4863 \n",
       "L 2375 4863 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-28\" d=\"M 1984 4856 \n",
       "Q 1566 4138 1362 3434 \n",
       "Q 1159 2731 1159 2009 \n",
       "Q 1159 1288 1364 580 \n",
       "Q 1569 -128 1984 -844 \n",
       "L 1484 -844 \n",
       "Q 1016 -109 783 600 \n",
       "Q 550 1309 550 2009 \n",
       "Q 550 2706 781 3412 \n",
       "Q 1013 4119 1484 4856 \n",
       "L 1984 4856 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-29\" d=\"M 513 4856 \n",
       "L 1013 4856 \n",
       "Q 1481 4119 1714 3412 \n",
       "Q 1947 2706 1947 2009 \n",
       "Q 1947 1309 1714 600 \n",
       "Q 1481 -109 1013 -844 \n",
       "L 513 -844 \n",
       "Q 928 -128 1133 580 \n",
       "Q 1338 1288 1338 2009 \n",
       "Q 1338 2731 1133 3434 \n",
       "Q 928 4138 513 4856 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "      </defs>\n",
       "      <use xlink:href=\"#DejaVuSans-66\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-28\" x=\"35.205078\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-78\" x=\"74.21875\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-29\" x=\"133.398438\"/>\n",
       "     </g>\n",
       "    </g>\n",
       "   </g>\n",
       "   <g id=\"line2d_21\">\n",
       "    <path d=\"M 62.205398 16.356404 \n",
       "L 66.778997 28.758673 \n",
       "L 71.25223 40.254351 \n",
       "L 75.625103 50.885672 \n",
       "L 79.89762 60.693932 \n",
       "L 84.069766 69.71943 \n",
       "L 88.141557 78.001586 \n",
       "L 92.098652 85.552371 \n",
       "L 95.955382 92.439193 \n",
       "L 99.711756 98.698485 \n",
       "L 103.367764 104.365689 \n",
       "L 106.937749 109.495131 \n",
       "L 110.407374 114.097459 \n",
       "L 113.790972 118.222147 \n",
       "L 117.088551 121.896578 \n",
       "L 120.300104 125.147432 \n",
       "L 123.439968 128.013076 \n",
       "L 126.493813 130.503685 \n",
       "L 129.490307 132.663297 \n",
       "L 132.415114 134.499735 \n",
       "L 135.282572 136.039782 \n",
       "L 138.092681 137.298905 \n",
       "L 140.859778 138.296815 \n",
       "L 143.583863 139.044708 \n",
       "L 146.279274 139.555719 \n",
       "L 148.960347 139.838031 \n",
       "L 151.627083 139.895257 \n",
       "L 154.293819 139.729507 \n",
       "L 156.960556 139.340782 \n",
       "L 159.641629 138.725189 \n",
       "L 162.351377 137.874004 \n",
       "L 165.0898 136.779918 \n",
       "L 167.871235 135.427954 \n",
       "L 170.710018 133.797992 \n",
       "L 173.591814 131.884886 \n",
       "L 176.545295 129.654006 \n",
       "L 179.556127 127.098282 \n",
       "L 182.638644 124.187247 \n",
       "L 185.792849 120.900114 \n",
       "L 189.033074 117.198512 \n",
       "L 192.359325 113.056218 \n",
       "L 195.757267 108.466448 \n",
       "L 199.255561 103.362913 \n",
       "L 202.839884 97.735889 \n",
       "L 206.524573 91.531403 \n",
       "L 210.309617 84.714684 \n",
       "L 214.166346 77.306827 \n",
       "L 218.123436 69.221459 \n",
       "L 222.195227 60.389213 \n",
       "L 226.353043 50.833935 \n",
       "L 230.61122 40.486193 \n",
       "L 234.969758 29.305793 \n",
       "L 239.428651 17.251603 \n",
       "L 239.744073 16.375279 \n",
       "L 239.744073 16.375279 \n",
       "\" clip-path=\"url(#p5e3e8d099e)\" style=\"fill: none; stroke: #1f77b4; stroke-width: 1.5; stroke-linecap: square\"/>\n",
       "   </g>\n",
       "   <g id=\"line2d_22\">\n",
       "    <path d=\"M 165.315417 136.679293 \n",
       "L 133.773377 135.261364 \n",
       "L 171.623825 133.219546 \n",
       "L 126.203288 130.279329 \n",
       "L 180.707933 126.045415 \n",
       "L 115.302359 119.94858 \n",
       "L 193.789047 111.169137 \n",
       "L 99.605021 98.526739 \n",
       "L 212.625853 80.321686 \n",
       "L 77.000855 54.10641 \n",
       "L 239.750852 16.356413 \n",
       "\" clip-path=\"url(#p5e3e8d099e)\" style=\"fill: none; stroke: #ff7f0e; stroke-width: 1.5; stroke-linecap: square\"/>\n",
       "    <defs>\n",
       "     <path id=\"m907449e165\" d=\"M 0 3 \n",
       "C 0.795609 3 1.55874 2.683901 2.12132 2.12132 \n",
       "C 2.683901 1.55874 3 0.795609 3 0 \n",
       "C 3 -0.795609 2.683901 -1.55874 2.12132 -2.12132 \n",
       "C 1.55874 -2.683901 0.795609 -3 0 -3 \n",
       "C -0.795609 -3 -1.55874 -2.683901 -2.12132 -2.12132 \n",
       "C -2.683901 -1.55874 -3 -0.795609 -3 0 \n",
       "C -3 0.795609 -2.683901 1.55874 -2.12132 2.12132 \n",
       "C -1.55874 2.683901 -0.795609 3 0 3 \n",
       "z\n",
       "\" style=\"stroke: #ff7f0e\"/>\n",
       "    </defs>\n",
       "    <g clip-path=\"url(#p5e3e8d099e)\">\n",
       "     <use xlink:href=\"#m907449e165\" x=\"165.315417\" y=\"136.679293\" style=\"fill: #ff7f0e; stroke: #ff7f0e\"/>\n",
       "     <use xlink:href=\"#m907449e165\" x=\"133.773377\" y=\"135.261364\" style=\"fill: #ff7f0e; stroke: #ff7f0e\"/>\n",
       "     <use xlink:href=\"#m907449e165\" x=\"171.623825\" y=\"133.219546\" style=\"fill: #ff7f0e; stroke: #ff7f0e\"/>\n",
       "     <use xlink:href=\"#m907449e165\" x=\"126.203288\" y=\"130.279329\" style=\"fill: #ff7f0e; stroke: #ff7f0e\"/>\n",
       "     <use xlink:href=\"#m907449e165\" x=\"180.707933\" y=\"126.045415\" style=\"fill: #ff7f0e; stroke: #ff7f0e\"/>\n",
       "     <use xlink:href=\"#m907449e165\" x=\"115.302359\" y=\"119.94858\" style=\"fill: #ff7f0e; stroke: #ff7f0e\"/>\n",
       "     <use xlink:href=\"#m907449e165\" x=\"193.789047\" y=\"111.169137\" style=\"fill: #ff7f0e; stroke: #ff7f0e\"/>\n",
       "     <use xlink:href=\"#m907449e165\" x=\"99.605021\" y=\"98.526739\" style=\"fill: #ff7f0e; stroke: #ff7f0e\"/>\n",
       "     <use xlink:href=\"#m907449e165\" x=\"212.625853\" y=\"80.321686\" style=\"fill: #ff7f0e; stroke: #ff7f0e\"/>\n",
       "     <use xlink:href=\"#m907449e165\" x=\"77.000855\" y=\"54.10641\" style=\"fill: #ff7f0e; stroke: #ff7f0e\"/>\n",
       "     <use xlink:href=\"#m907449e165\" x=\"239.750852\" y=\"16.356413\" style=\"fill: #ff7f0e; stroke: #ff7f0e\"/>\n",
       "    </g>\n",
       "   </g>\n",
       "   <g id=\"patch_3\">\n",
       "    <path d=\"M 53.328125 146.079132 \n",
       "L 53.328125 10.179132 \n",
       "\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_4\">\n",
       "    <path d=\"M 248.628125 146.079132 \n",
       "L 248.628125 10.179132 \n",
       "\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_5\">\n",
       "    <path d=\"M 53.328125 146.079132 \n",
       "L 248.628125 146.079132 \n",
       "\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_6\">\n",
       "    <path d=\"M 53.328125 10.179132 \n",
       "L 248.628125 10.179132 \n",
       "\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n",
       "   </g>\n",
       "  </g>\n",
       " </g>\n",
       " <defs>\n",
       "  <clipPath id=\"p5e3e8d099e\">\n",
       "   <rect x=\"53.328125\" y=\"10.179132\" width=\"195.3\" height=\"135.9\"/>\n",
       "  </clipPath>\n",
       " </defs>\n",
       "</svg>\n"
      ],
      "text/plain": [
       "<Figure size 252x180 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "show_trace(gd(1.1, f_grad), f)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f6918593",
   "metadata": {
    "origin_pos": 16
   },
   "source": [
    "### 局部最小值\n",
    "\n",
    "为了演示非凸函数的梯度下降，考虑函数$f(x) = x \\cdot \\cos(cx)$，其中$c$为某常数。\n",
    "这个函数有无穷多个局部最小值。\n",
    "根据我们选择的学习率，我们最终可能只会得到许多解的一个。\n",
    "下面的例子说明了（不切实际的）高学习率如何导致较差的局部最小值。\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "51583e82",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-08-18T07:01:27.356773Z",
     "iopub.status.busy": "2023-08-18T07:01:27.355855Z",
     "iopub.status.idle": "2023-08-18T07:01:27.649547Z",
     "shell.execute_reply": "2023-08-18T07:01:27.648346Z"
    },
    "origin_pos": 17,
    "tab": [
     "pytorch"
    ]
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "epoch 10, x: -1.528166\n"
     ]
    },
    {
     "data": {
      "image/svg+xml": [
       "<?xml version=\"1.0\" encoding=\"utf-8\" standalone=\"no\"?>\n",
       "<!DOCTYPE svg PUBLIC \"-//W3C//DTD SVG 1.1//EN\"\n",
       "  \"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd\">\n",
       "<svg xmlns:xlink=\"http://www.w3.org/1999/xlink\" width=\"245.120313pt\" height=\"180.65625pt\" viewBox=\"0 0 245.120313 180.65625\" xmlns=\"http://www.w3.org/2000/svg\" version=\"1.1\">\n",
       " <metadata>\n",
       "  <rdf:RDF xmlns:dc=\"http://purl.org/dc/elements/1.1/\" xmlns:cc=\"http://creativecommons.org/ns#\" xmlns:rdf=\"http://www.w3.org/1999/02/22-rdf-syntax-ns#\">\n",
       "   <cc:Work>\n",
       "    <dc:type rdf:resource=\"http://purl.org/dc/dcmitype/StillImage\"/>\n",
       "    <dc:date>2023-08-18T07:01:27.605331</dc:date>\n",
       "    <dc:format>image/svg+xml</dc:format>\n",
       "    <dc:creator>\n",
       "     <cc:Agent>\n",
       "      <dc:title>Matplotlib v3.5.1, https://matplotlib.org/</dc:title>\n",
       "     </cc:Agent>\n",
       "    </dc:creator>\n",
       "   </cc:Work>\n",
       "  </rdf:RDF>\n",
       " </metadata>\n",
       " <defs>\n",
       "  <style type=\"text/css\">*{stroke-linejoin: round; stroke-linecap: butt}</style>\n",
       " </defs>\n",
       " <g id=\"figure_1\">\n",
       "  <g id=\"patch_1\">\n",
       "   <path d=\"M 0 180.65625 \n",
       "L 245.120313 180.65625 \n",
       "L 245.120313 0 \n",
       "L 0 0 \n",
       "L 0 180.65625 \n",
       "z\n",
       "\" style=\"fill: none\"/>\n",
       "  </g>\n",
       "  <g id=\"axes_1\">\n",
       "   <g id=\"patch_2\">\n",
       "    <path d=\"M 42.620312 143.1 \n",
       "L 237.920313 143.1 \n",
       "L 237.920313 7.2 \n",
       "L 42.620312 7.2 \n",
       "z\n",
       "\" style=\"fill: #ffffff\"/>\n",
       "   </g>\n",
       "   <g id=\"matplotlib.axis_1\">\n",
       "    <g id=\"xtick_1\">\n",
       "     <g id=\"line2d_1\">\n",
       "      <path d=\"M 51.497585 143.1 \n",
       "L 51.497585 7.2 \n",
       "\" clip-path=\"url(#p179f62d71c)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_2\">\n",
       "      <defs>\n",
       "       <path id=\"me00d650fad\" d=\"M 0 0 \n",
       "L 0 3.5 \n",
       "\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </defs>\n",
       "      <g>\n",
       "       <use xlink:href=\"#me00d650fad\" x=\"51.497585\" y=\"143.1\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_1\">\n",
       "      <!-- −10 -->\n",
       "      <g transform=\"translate(40.945241 157.698438)scale(0.1 -0.1)\">\n",
       "       <defs>\n",
       "        <path id=\"DejaVuSans-2212\" d=\"M 678 2272 \n",
       "L 4684 2272 \n",
       "L 4684 1741 \n",
       "L 678 1741 \n",
       "L 678 2272 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "        <path id=\"DejaVuSans-31\" d=\"M 794 531 \n",
       "L 1825 531 \n",
       "L 1825 4091 \n",
       "L 703 3866 \n",
       "L 703 4441 \n",
       "L 1819 4666 \n",
       "L 2450 4666 \n",
       "L 2450 531 \n",
       "L 3481 531 \n",
       "L 3481 0 \n",
       "L 794 0 \n",
       "L 794 531 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "        <path id=\"DejaVuSans-30\" d=\"M 2034 4250 \n",
       "Q 1547 4250 1301 3770 \n",
       "Q 1056 3291 1056 2328 \n",
       "Q 1056 1369 1301 889 \n",
       "Q 1547 409 2034 409 \n",
       "Q 2525 409 2770 889 \n",
       "Q 3016 1369 3016 2328 \n",
       "Q 3016 3291 2770 3770 \n",
       "Q 2525 4250 2034 4250 \n",
       "z\n",
       "M 2034 4750 \n",
       "Q 2819 4750 3233 4129 \n",
       "Q 3647 3509 3647 2328 \n",
       "Q 3647 1150 3233 529 \n",
       "Q 2819 -91 2034 -91 \n",
       "Q 1250 -91 836 529 \n",
       "Q 422 1150 422 2328 \n",
       "Q 422 3509 836 4129 \n",
       "Q 1250 4750 2034 4750 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       </defs>\n",
       "       <use xlink:href=\"#DejaVuSans-2212\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-31\" x=\"83.789062\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"147.412109\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_2\">\n",
       "     <g id=\"line2d_3\">\n",
       "      <path d=\"M 95.883949 143.1 \n",
       "L 95.883949 7.2 \n",
       "\" clip-path=\"url(#p179f62d71c)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_4\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#me00d650fad\" x=\"95.883949\" y=\"143.1\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_2\">\n",
       "      <!-- −5 -->\n",
       "      <g transform=\"translate(88.512855 157.698438)scale(0.1 -0.1)\">\n",
       "       <defs>\n",
       "        <path id=\"DejaVuSans-35\" d=\"M 691 4666 \n",
       "L 3169 4666 \n",
       "L 3169 4134 \n",
       "L 1269 4134 \n",
       "L 1269 2991 \n",
       "Q 1406 3038 1543 3061 \n",
       "Q 1681 3084 1819 3084 \n",
       "Q 2600 3084 3056 2656 \n",
       "Q 3513 2228 3513 1497 \n",
       "Q 3513 744 3044 326 \n",
       "Q 2575 -91 1722 -91 \n",
       "Q 1428 -91 1123 -41 \n",
       "Q 819 9 494 109 \n",
       "L 494 744 \n",
       "Q 775 591 1075 516 \n",
       "Q 1375 441 1709 441 \n",
       "Q 2250 441 2565 725 \n",
       "Q 2881 1009 2881 1497 \n",
       "Q 2881 1984 2565 2268 \n",
       "Q 2250 2553 1709 2553 \n",
       "Q 1456 2553 1204 2497 \n",
       "Q 953 2441 691 2322 \n",
       "L 691 4666 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       </defs>\n",
       "       <use xlink:href=\"#DejaVuSans-2212\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-35\" x=\"83.789062\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_3\">\n",
       "     <g id=\"line2d_5\">\n",
       "      <path d=\"M 140.270312 143.1 \n",
       "L 140.270312 7.2 \n",
       "\" clip-path=\"url(#p179f62d71c)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_6\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#me00d650fad\" x=\"140.270312\" y=\"143.1\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_3\">\n",
       "      <!-- 0 -->\n",
       "      <g transform=\"translate(137.089062 157.698438)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#DejaVuSans-30\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_4\">\n",
       "     <g id=\"line2d_7\">\n",
       "      <path d=\"M 184.656676 143.1 \n",
       "L 184.656676 7.2 \n",
       "\" clip-path=\"url(#p179f62d71c)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_8\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#me00d650fad\" x=\"184.656676\" y=\"143.1\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_4\">\n",
       "      <!-- 5 -->\n",
       "      <g transform=\"translate(181.475426 157.698438)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#DejaVuSans-35\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_5\">\n",
       "     <g id=\"line2d_9\">\n",
       "      <path d=\"M 229.04304 143.1 \n",
       "L 229.04304 7.2 \n",
       "\" clip-path=\"url(#p179f62d71c)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_10\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#me00d650fad\" x=\"229.04304\" y=\"143.1\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_5\">\n",
       "      <!-- 10 -->\n",
       "      <g transform=\"translate(222.68054 157.698438)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#DejaVuSans-31\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"63.623047\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"text_6\">\n",
       "     <!-- x -->\n",
       "     <g transform=\"translate(137.310937 171.376563)scale(0.1 -0.1)\">\n",
       "      <defs>\n",
       "       <path id=\"DejaVuSans-78\" d=\"M 3513 3500 \n",
       "L 2247 1797 \n",
       "L 3578 0 \n",
       "L 2900 0 \n",
       "L 1881 1375 \n",
       "L 863 0 \n",
       "L 184 0 \n",
       "L 1544 1831 \n",
       "L 300 3500 \n",
       "L 978 3500 \n",
       "L 1906 2253 \n",
       "L 2834 3500 \n",
       "L 3513 3500 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "      </defs>\n",
       "      <use xlink:href=\"#DejaVuSans-78\"/>\n",
       "     </g>\n",
       "    </g>\n",
       "   </g>\n",
       "   <g id=\"matplotlib.axis_2\">\n",
       "    <g id=\"ytick_1\">\n",
       "     <g id=\"line2d_11\">\n",
       "      <path d=\"M 42.620312 119.411597 \n",
       "L 237.920313 119.411597 \n",
       "\" clip-path=\"url(#p179f62d71c)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_12\">\n",
       "      <defs>\n",
       "       <path id=\"m91884548f2\" d=\"M 0 0 \n",
       "L -3.5 0 \n",
       "\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </defs>\n",
       "      <g>\n",
       "       <use xlink:href=\"#m91884548f2\" x=\"42.620312\" y=\"119.411597\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_7\">\n",
       "      <!-- −5 -->\n",
       "      <g transform=\"translate(20.878125 123.210816)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#DejaVuSans-2212\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-35\" x=\"83.789062\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_2\">\n",
       "     <g id=\"line2d_13\">\n",
       "      <path d=\"M 42.620312 75.15 \n",
       "L 237.920313 75.15 \n",
       "\" clip-path=\"url(#p179f62d71c)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_14\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#m91884548f2\" x=\"42.620312\" y=\"75.15\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_8\">\n",
       "      <!-- 0 -->\n",
       "      <g transform=\"translate(29.257812 78.949219)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#DejaVuSans-30\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_3\">\n",
       "     <g id=\"line2d_15\">\n",
       "      <path d=\"M 42.620312 30.888403 \n",
       "L 237.920313 30.888403 \n",
       "\" clip-path=\"url(#p179f62d71c)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_16\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#m91884548f2\" x=\"42.620312\" y=\"30.888403\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_9\">\n",
       "      <!-- 5 -->\n",
       "      <g transform=\"translate(29.257812 34.687622)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#DejaVuSans-35\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"text_10\">\n",
       "     <!-- f(x) -->\n",
       "     <g transform=\"translate(14.798437 83.771094)rotate(-90)scale(0.1 -0.1)\">\n",
       "      <defs>\n",
       "       <path id=\"DejaVuSans-66\" d=\"M 2375 4863 \n",
       "L 2375 4384 \n",
       "L 1825 4384 \n",
       "Q 1516 4384 1395 4259 \n",
       "Q 1275 4134 1275 3809 \n",
       "L 1275 3500 \n",
       "L 2222 3500 \n",
       "L 2222 3053 \n",
       "L 1275 3053 \n",
       "L 1275 0 \n",
       "L 697 0 \n",
       "L 697 3053 \n",
       "L 147 3053 \n",
       "L 147 3500 \n",
       "L 697 3500 \n",
       "L 697 3744 \n",
       "Q 697 4328 969 4595 \n",
       "Q 1241 4863 1831 4863 \n",
       "L 2375 4863 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-28\" d=\"M 1984 4856 \n",
       "Q 1566 4138 1362 3434 \n",
       "Q 1159 2731 1159 2009 \n",
       "Q 1159 1288 1364 580 \n",
       "Q 1569 -128 1984 -844 \n",
       "L 1484 -844 \n",
       "Q 1016 -109 783 600 \n",
       "Q 550 1309 550 2009 \n",
       "Q 550 2706 781 3412 \n",
       "Q 1013 4119 1484 4856 \n",
       "L 1984 4856 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-29\" d=\"M 513 4856 \n",
       "L 1013 4856 \n",
       "Q 1481 4119 1714 3412 \n",
       "Q 1947 2706 1947 2009 \n",
       "Q 1947 1309 1714 600 \n",
       "Q 1481 -109 1013 -844 \n",
       "L 513 -844 \n",
       "Q 928 -128 1133 580 \n",
       "Q 1338 1288 1338 2009 \n",
       "Q 1338 2731 1133 3434 \n",
       "Q 928 4138 513 4856 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "      </defs>\n",
       "      <use xlink:href=\"#DejaVuSans-66\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-28\" x=\"35.205078\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-78\" x=\"74.21875\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-29\" x=\"133.398438\"/>\n",
       "     </g>\n",
       "    </g>\n",
       "   </g>\n",
       "   <g id=\"line2d_17\">\n",
       "    <path d=\"M 51.497585 75.150001 \n",
       "L 54.515859 61.507494 \n",
       "L 57.090268 50.885046 \n",
       "L 59.309586 42.62211 \n",
       "L 61.351355 35.837137 \n",
       "L 63.215583 30.377167 \n",
       "L 64.991044 25.861634 \n",
       "L 66.588956 22.386195 \n",
       "L 68.098084 19.624675 \n",
       "L 69.518451 17.490893 \n",
       "L 70.850038 15.899777 \n",
       "L 72.092859 14.769329 \n",
       "L 73.246906 14.022221 \n",
       "L 74.312178 13.586851 \n",
       "L 75.377449 13.390552 \n",
       "L 76.442725 13.427546 \n",
       "L 77.507997 13.691167 \n",
       "L 78.662039 14.22383 \n",
       "L 79.816086 15.003166 \n",
       "L 81.058907 16.105109 \n",
       "L 82.479269 17.678382 \n",
       "L 83.988402 19.689963 \n",
       "L 85.675089 22.313917 \n",
       "L 87.628087 25.787156 \n",
       "L 89.847405 30.206604 \n",
       "L 92.599359 36.214766 \n",
       "L 96.59413 45.535707 \n",
       "L 102.719453 59.79435 \n",
       "L 105.648948 66.04003 \n",
       "L 108.134584 70.834186 \n",
       "L 110.353902 74.634558 \n",
       "L 112.306903 77.556075 \n",
       "L 114.17113 79.946414 \n",
       "L 115.857812 81.757913 \n",
       "L 117.45572 83.158273 \n",
       "L 118.964857 84.195921 \n",
       "L 120.385222 84.920387 \n",
       "L 121.805586 85.403652 \n",
       "L 123.225949 85.651421 \n",
       "L 124.646313 85.67174 \n",
       "L 126.066676 85.474901 \n",
       "L 127.575812 85.041746 \n",
       "L 129.173722 84.350543 \n",
       "L 130.949177 83.330131 \n",
       "L 132.902176 81.942544 \n",
       "L 135.210267 80.014888 \n",
       "L 138.22854 77.174086 \n",
       "L 146.04054 69.663818 \n",
       "L 148.34863 67.823794 \n",
       "L 150.301631 66.531906 \n",
       "L 152.077085 65.614112 \n",
       "L 153.674995 65.026673 \n",
       "L 155.18413 64.700263 \n",
       "L 156.604495 64.61058 \n",
       "L 158.024858 64.743475 \n",
       "L 159.445222 65.10813 \n",
       "L 160.865585 65.711433 \n",
       "L 162.285949 66.557882 \n",
       "L 163.795086 67.725879 \n",
       "L 165.392994 69.263456 \n",
       "L 167.079676 71.216922 \n",
       "L 168.855131 73.628385 \n",
       "L 170.808131 76.680042 \n",
       "L 172.938677 80.448158 \n",
       "L 175.33554 85.168809 \n",
       "L 178.087492 91.099238 \n",
       "L 181.815947 99.711378 \n",
       "L 189.272858 117.052872 \n",
       "L 191.936041 122.62392 \n",
       "L 194.155359 126.773504 \n",
       "L 196.019583 129.822238 \n",
       "L 197.706269 132.178722 \n",
       "L 199.215402 133.924756 \n",
       "L 200.635769 135.226991 \n",
       "L 201.878586 136.07617 \n",
       "L 203.032632 136.608833 \n",
       "L 204.097904 136.872454 \n",
       "L 205.16318 136.909448 \n",
       "L 206.228452 136.713149 \n",
       "L 207.293723 136.277775 \n",
       "L 208.358995 135.598428 \n",
       "L 209.513041 134.582583 \n",
       "L 210.667088 133.272041 \n",
       "L 211.909901 131.528413 \n",
       "L 213.241497 129.276672 \n",
       "L 214.661859 126.438757 \n",
       "L 216.259771 122.713826 \n",
       "L 217.946449 118.183227 \n",
       "L 219.721902 112.772058 \n",
       "L 221.674904 106.096761 \n",
       "L 223.805448 98.012828 \n",
       "L 226.202315 88.022895 \n",
       "L 228.954265 75.566753 \n",
       "L 228.954265 75.566753 \n",
       "\" clip-path=\"url(#p179f62d71c)\" style=\"fill: none; stroke: #1f77b4; stroke-width: 1.5; stroke-linecap: square\"/>\n",
       "   </g>\n",
       "   <g id=\"line2d_18\">\n",
       "    <path d=\"M 229.04304 75.149999 \n",
       "L 145.376715 70.243883 \n",
       "L 129.559103 84.150541 \n",
       "L 120.033699 84.763719 \n",
       "L 128.344442 84.73779 \n",
       "L 120.680218 85.040448 \n",
       "L 127.613196 85.028227 \n",
       "L 121.143037 85.207934 \n",
       "L 127.097544 85.203029 \n",
       "L 121.500618 85.319975 \n",
       "L 126.704368 85.319115 \n",
       "\" clip-path=\"url(#p179f62d71c)\" style=\"fill: none; stroke: #ff7f0e; stroke-width: 1.5; stroke-linecap: square\"/>\n",
       "    <defs>\n",
       "     <path id=\"mc9bbaa1c85\" d=\"M 0 3 \n",
       "C 0.795609 3 1.55874 2.683901 2.12132 2.12132 \n",
       "C 2.683901 1.55874 3 0.795609 3 0 \n",
       "C 3 -0.795609 2.683901 -1.55874 2.12132 -2.12132 \n",
       "C 1.55874 -2.683901 0.795609 -3 0 -3 \n",
       "C -0.795609 -3 -1.55874 -2.683901 -2.12132 -2.12132 \n",
       "C -2.683901 -1.55874 -3 -0.795609 -3 0 \n",
       "C -3 0.795609 -2.683901 1.55874 -2.12132 2.12132 \n",
       "C -1.55874 2.683901 -0.795609 3 0 3 \n",
       "z\n",
       "\" style=\"stroke: #ff7f0e\"/>\n",
       "    </defs>\n",
       "    <g clip-path=\"url(#p179f62d71c)\">\n",
       "     <use xlink:href=\"#mc9bbaa1c85\" x=\"229.04304\" y=\"75.149999\" style=\"fill: #ff7f0e; stroke: #ff7f0e\"/>\n",
       "     <use xlink:href=\"#mc9bbaa1c85\" x=\"145.376715\" y=\"70.243883\" style=\"fill: #ff7f0e; stroke: #ff7f0e\"/>\n",
       "     <use xlink:href=\"#mc9bbaa1c85\" x=\"129.559103\" y=\"84.150541\" style=\"fill: #ff7f0e; stroke: #ff7f0e\"/>\n",
       "     <use xlink:href=\"#mc9bbaa1c85\" x=\"120.033699\" y=\"84.763719\" style=\"fill: #ff7f0e; stroke: #ff7f0e\"/>\n",
       "     <use xlink:href=\"#mc9bbaa1c85\" x=\"128.344442\" y=\"84.73779\" style=\"fill: #ff7f0e; stroke: #ff7f0e\"/>\n",
       "     <use xlink:href=\"#mc9bbaa1c85\" x=\"120.680218\" y=\"85.040448\" style=\"fill: #ff7f0e; stroke: #ff7f0e\"/>\n",
       "     <use xlink:href=\"#mc9bbaa1c85\" x=\"127.613196\" y=\"85.028227\" style=\"fill: #ff7f0e; stroke: #ff7f0e\"/>\n",
       "     <use xlink:href=\"#mc9bbaa1c85\" x=\"121.143037\" y=\"85.207934\" style=\"fill: #ff7f0e; stroke: #ff7f0e\"/>\n",
       "     <use xlink:href=\"#mc9bbaa1c85\" x=\"127.097544\" y=\"85.203029\" style=\"fill: #ff7f0e; stroke: #ff7f0e\"/>\n",
       "     <use xlink:href=\"#mc9bbaa1c85\" x=\"121.500618\" y=\"85.319975\" style=\"fill: #ff7f0e; stroke: #ff7f0e\"/>\n",
       "     <use xlink:href=\"#mc9bbaa1c85\" x=\"126.704368\" y=\"85.319115\" style=\"fill: #ff7f0e; stroke: #ff7f0e\"/>\n",
       "    </g>\n",
       "   </g>\n",
       "   <g id=\"patch_3\">\n",
       "    <path d=\"M 42.620312 143.1 \n",
       "L 42.620312 7.2 \n",
       "\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_4\">\n",
       "    <path d=\"M 237.920313 143.1 \n",
       "L 237.920313 7.2 \n",
       "\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_5\">\n",
       "    <path d=\"M 42.620312 143.1 \n",
       "L 237.920313 143.1 \n",
       "\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_6\">\n",
       "    <path d=\"M 42.620312 7.2 \n",
       "L 237.920313 7.2 \n",
       "\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n",
       "   </g>\n",
       "  </g>\n",
       " </g>\n",
       " <defs>\n",
       "  <clipPath id=\"p179f62d71c\">\n",
       "   <rect x=\"42.620312\" y=\"7.2\" width=\"195.3\" height=\"135.9\"/>\n",
       "  </clipPath>\n",
       " </defs>\n",
       "</svg>\n"
      ],
      "text/plain": [
       "<Figure size 252x180 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "c = torch.tensor(0.15 * np.pi)\n",
    "\n",
    "def f(x):  # 目标函数\n",
    "    return x * torch.cos(c * x)\n",
    "\n",
    "def f_grad(x):  # 目标函数的梯度\n",
    "    return torch.cos(c * x) - c * x * torch.sin(c * x)\n",
    "\n",
    "show_trace(gd(2, f_grad), f)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3a19f7d8",
   "metadata": {
    "origin_pos": 18
   },
   "source": [
    "## 多元梯度下降\n",
    "\n",
    "现在我们对单变量的情况有了更好的理解，让我们考虑一下$\\mathbf{x} = [x_1, x_2, \\ldots, x_d]^\\top$的情况。\n",
    "即目标函数$f: \\mathbb{R}^d \\to \\mathbb{R}$将向量映射成标量。\n",
    "相应地，它的梯度也是多元的，它是一个由$d$个偏导数组成的向量：\n",
    "\n",
    "$$\\nabla f(\\mathbf{x}) = \\bigg[\\frac{\\partial f(\\mathbf{x})}{\\partial x_1}, \\frac{\\partial f(\\mathbf{x})}{\\partial x_2}, \\ldots, \\frac{\\partial f(\\mathbf{x})}{\\partial x_d}\\bigg]^\\top.$$\n",
    "\n",
    "梯度中的每个偏导数元素$\\partial f(\\mathbf{x})/\\partial x_i$代表了当输入$x_i$时$f$在$\\mathbf{x}$处的变化率。\n",
    "和先前单变量的情况一样，我们可以对多变量函数使用相应的泰勒近似来思考。\n",
    "具体来说，\n",
    "\n",
    "$$f(\\mathbf{x} + \\boldsymbol{\\epsilon}) = f(\\mathbf{x}) + \\mathbf{\\boldsymbol{\\epsilon}}^\\top \\nabla f(\\mathbf{x}) + \\mathcal{O}(\\|\\boldsymbol{\\epsilon}\\|^2).$$\n",
    ":eqlabel:`gd-multi-taylor`\n",
    "\n",
    "换句话说，在$\\boldsymbol{\\epsilon}$的二阶项中，\n",
    "最陡下降的方向由负梯度$-\\nabla f(\\mathbf{x})$得出。\n",
    "选择合适的学习率$\\eta > 0$来生成典型的梯度下降算法：\n",
    "\n",
    "$$\\mathbf{x} \\leftarrow \\mathbf{x} - \\eta \\nabla f(\\mathbf{x}).$$\n",
    "\n",
    "这个算法在实践中的表现如何呢？\n",
    "我们构造一个目标函数$f(\\mathbf{x})=x_1^2+2x_2^2$，\n",
    "并有二维向量$\\mathbf{x} = [x_1, x_2]^\\top$作为输入，\n",
    "标量作为输出。\n",
    "梯度由$\\nabla f(\\mathbf{x}) = [2x_1, 4x_2]^\\top$给出。\n",
    "我们将从初始位置$[-5, -2]$通过梯度下降观察$\\mathbf{x}$的轨迹。\n",
    "\n",
    "我们还需要两个辅助函数：\n",
    "第一个是update函数，并将其应用于初始值20次；\n",
    "第二个函数会显示$\\mathbf{x}$的轨迹。\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "ef6dbba5",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-08-18T07:01:27.655203Z",
     "iopub.status.busy": "2023-08-18T07:01:27.654471Z",
     "iopub.status.idle": "2023-08-18T07:01:27.663479Z",
     "shell.execute_reply": "2023-08-18T07:01:27.662393Z"
    },
    "origin_pos": 19,
    "tab": [
     "pytorch"
    ]
   },
   "outputs": [],
   "source": [
    "def train_2d(trainer, steps=20, f_grad=None):  #@save\n",
    "    \"\"\"用定制的训练机优化2D目标函数\"\"\"\n",
    "    # s1和s2是稍后将使用的内部状态变量\n",
    "    x1, x2, s1, s2 = -5, -2, 0, 0\n",
    "    results = [(x1, x2)]\n",
    "    for i in range(steps):\n",
    "        if f_grad:\n",
    "            x1, x2, s1, s2 = trainer(x1, x2, s1, s2, f_grad)\n",
    "        else:\n",
    "            x1, x2, s1, s2 = trainer(x1, x2, s1, s2)\n",
    "        results.append((x1, x2))\n",
    "    print(f'epoch {i + 1}, x1: {float(x1):f}, x2: {float(x2):f}')\n",
    "    return results"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "d47839ef",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-08-18T07:01:27.668397Z",
     "iopub.status.busy": "2023-08-18T07:01:27.667552Z",
     "iopub.status.idle": "2023-08-18T07:01:27.675730Z",
     "shell.execute_reply": "2023-08-18T07:01:27.674619Z"
    },
    "origin_pos": 21,
    "tab": [
     "pytorch"
    ]
   },
   "outputs": [],
   "source": [
    "def show_trace_2d(f, results):  #@save\n",
    "    \"\"\"显示优化过程中2D变量的轨迹\"\"\"\n",
    "    d2l.set_figsize()\n",
    "    d2l.plt.plot(*zip(*results), '-o', color='#ff7f0e')\n",
    "    x1, x2 = torch.meshgrid(torch.arange(-5.5, 1.0, 0.1),\n",
    "                          torch.arange(-3.0, 1.0, 0.1), indexing='ij')\n",
    "    d2l.plt.contour(x1, x2, f(x1, x2), colors='#1f77b4')\n",
    "    d2l.plt.xlabel('x1')\n",
    "    d2l.plt.ylabel('x2')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bd6785e8",
   "metadata": {
    "origin_pos": 23
   },
   "source": [
    "接下来，我们观察学习率$\\eta = 0.1$时优化变量$\\mathbf{x}$的轨迹。\n",
    "可以看到，经过20步之后，$\\mathbf{x}$的值接近其位于$[0, 0]$的最小值。\n",
    "虽然进展相当顺利，但相当缓慢。\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "6d58435d",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-08-18T07:01:27.679558Z",
     "iopub.status.busy": "2023-08-18T07:01:27.679151Z",
     "iopub.status.idle": "2023-08-18T07:01:27.882280Z",
     "shell.execute_reply": "2023-08-18T07:01:27.881035Z"
    },
    "origin_pos": 24,
    "tab": [
     "pytorch"
    ]
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "epoch 20, x1: -0.057646, x2: -0.000073\n"
     ]
    },
    {
     "data": {
      "image/svg+xml": [
       "<?xml version=\"1.0\" encoding=\"utf-8\" standalone=\"no\"?>\n",
       "<!DOCTYPE svg PUBLIC \"-//W3C//DTD SVG 1.1//EN\"\n",
       "  \"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd\">\n",
       "<svg xmlns:xlink=\"http://www.w3.org/1999/xlink\" width=\"245.120313pt\" height=\"180.65625pt\" viewBox=\"0 0 245.120313 180.65625\" xmlns=\"http://www.w3.org/2000/svg\" version=\"1.1\">\n",
       " <metadata>\n",
       "  <rdf:RDF xmlns:dc=\"http://purl.org/dc/elements/1.1/\" xmlns:cc=\"http://creativecommons.org/ns#\" xmlns:rdf=\"http://www.w3.org/1999/02/22-rdf-syntax-ns#\">\n",
       "   <cc:Work>\n",
       "    <dc:type rdf:resource=\"http://purl.org/dc/dcmitype/StillImage\"/>\n",
       "    <dc:date>2023-08-18T07:01:27.838043</dc:date>\n",
       "    <dc:format>image/svg+xml</dc:format>\n",
       "    <dc:creator>\n",
       "     <cc:Agent>\n",
       "      <dc:title>Matplotlib v3.5.1, https://matplotlib.org/</dc:title>\n",
       "     </cc:Agent>\n",
       "    </dc:creator>\n",
       "   </cc:Work>\n",
       "  </rdf:RDF>\n",
       " </metadata>\n",
       " <defs>\n",
       "  <style type=\"text/css\">*{stroke-linejoin: round; stroke-linecap: butt}</style>\n",
       " </defs>\n",
       " <g id=\"figure_1\">\n",
       "  <g id=\"patch_1\">\n",
       "   <path d=\"M 0 180.65625 \n",
       "L 245.120313 180.65625 \n",
       "L 245.120313 0 \n",
       "L 0 0 \n",
       "L 0 180.65625 \n",
       "z\n",
       "\" style=\"fill: none\"/>\n",
       "  </g>\n",
       "  <g id=\"axes_1\">\n",
       "   <g id=\"patch_2\">\n",
       "    <path d=\"M 42.620312 143.1 \n",
       "L 237.920313 143.1 \n",
       "L 237.920313 7.2 \n",
       "L 42.620312 7.2 \n",
       "z\n",
       "\" style=\"fill: #ffffff\"/>\n",
       "   </g>\n",
       "   <g id=\"matplotlib.axis_1\">\n",
       "    <g id=\"xtick_1\">\n",
       "     <g id=\"line2d_1\">\n",
       "      <defs>\n",
       "       <path id=\"m5e41782ad5\" d=\"M 0 0 \n",
       "L 0 3.5 \n",
       "\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </defs>\n",
       "      <g>\n",
       "       <use xlink:href=\"#m5e41782ad5\" x=\"88.39375\" y=\"143.1\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_1\">\n",
       "      <!-- −4 -->\n",
       "      <g transform=\"translate(81.022656 157.698438)scale(0.1 -0.1)\">\n",
       "       <defs>\n",
       "        <path id=\"DejaVuSans-2212\" d=\"M 678 2272 \n",
       "L 4684 2272 \n",
       "L 4684 1741 \n",
       "L 678 1741 \n",
       "L 678 2272 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "        <path id=\"DejaVuSans-34\" d=\"M 2419 4116 \n",
       "L 825 1625 \n",
       "L 2419 1625 \n",
       "L 2419 4116 \n",
       "z\n",
       "M 2253 4666 \n",
       "L 3047 4666 \n",
       "L 3047 1625 \n",
       "L 3713 1625 \n",
       "L 3713 1100 \n",
       "L 3047 1100 \n",
       "L 3047 0 \n",
       "L 2419 0 \n",
       "L 2419 1100 \n",
       "L 313 1100 \n",
       "L 313 1709 \n",
       "L 2253 4666 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       </defs>\n",
       "       <use xlink:href=\"#DejaVuSans-2212\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-34\" x=\"83.789062\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_2\">\n",
       "     <g id=\"line2d_2\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#m5e41782ad5\" x=\"149.425\" y=\"143.1\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_2\">\n",
       "      <!-- −2 -->\n",
       "      <g transform=\"translate(142.053907 157.698438)scale(0.1 -0.1)\">\n",
       "       <defs>\n",
       "        <path id=\"DejaVuSans-32\" d=\"M 1228 531 \n",
       "L 3431 531 \n",
       "L 3431 0 \n",
       "L 469 0 \n",
       "L 469 531 \n",
       "Q 828 903 1448 1529 \n",
       "Q 2069 2156 2228 2338 \n",
       "Q 2531 2678 2651 2914 \n",
       "Q 2772 3150 2772 3378 \n",
       "Q 2772 3750 2511 3984 \n",
       "Q 2250 4219 1831 4219 \n",
       "Q 1534 4219 1204 4116 \n",
       "Q 875 4013 500 3803 \n",
       "L 500 4441 \n",
       "Q 881 4594 1212 4672 \n",
       "Q 1544 4750 1819 4750 \n",
       "Q 2544 4750 2975 4387 \n",
       "Q 3406 4025 3406 3419 \n",
       "Q 3406 3131 3298 2873 \n",
       "Q 3191 2616 2906 2266 \n",
       "Q 2828 2175 2409 1742 \n",
       "Q 1991 1309 1228 531 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       </defs>\n",
       "       <use xlink:href=\"#DejaVuSans-2212\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-32\" x=\"83.789062\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_3\">\n",
       "     <g id=\"line2d_3\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#m5e41782ad5\" x=\"210.456251\" y=\"143.1\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_3\">\n",
       "      <!-- 0 -->\n",
       "      <g transform=\"translate(207.275001 157.698438)scale(0.1 -0.1)\">\n",
       "       <defs>\n",
       "        <path id=\"DejaVuSans-30\" d=\"M 2034 4250 \n",
       "Q 1547 4250 1301 3770 \n",
       "Q 1056 3291 1056 2328 \n",
       "Q 1056 1369 1301 889 \n",
       "Q 1547 409 2034 409 \n",
       "Q 2525 409 2770 889 \n",
       "Q 3016 1369 3016 2328 \n",
       "Q 3016 3291 2770 3770 \n",
       "Q 2525 4250 2034 4250 \n",
       "z\n",
       "M 2034 4750 \n",
       "Q 2819 4750 3233 4129 \n",
       "Q 3647 3509 3647 2328 \n",
       "Q 3647 1150 3233 529 \n",
       "Q 2819 -91 2034 -91 \n",
       "Q 1250 -91 836 529 \n",
       "Q 422 1150 422 2328 \n",
       "Q 422 3509 836 4129 \n",
       "Q 1250 4750 2034 4750 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       </defs>\n",
       "       <use xlink:href=\"#DejaVuSans-30\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"text_4\">\n",
       "     <!-- x1 -->\n",
       "     <g transform=\"translate(134.129687 171.376563)scale(0.1 -0.1)\">\n",
       "      <defs>\n",
       "       <path id=\"DejaVuSans-78\" d=\"M 3513 3500 \n",
       "L 2247 1797 \n",
       "L 3578 0 \n",
       "L 2900 0 \n",
       "L 1881 1375 \n",
       "L 863 0 \n",
       "L 184 0 \n",
       "L 1544 1831 \n",
       "L 300 3500 \n",
       "L 978 3500 \n",
       "L 1906 2253 \n",
       "L 2834 3500 \n",
       "L 3513 3500 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-31\" d=\"M 794 531 \n",
       "L 1825 531 \n",
       "L 1825 4091 \n",
       "L 703 3866 \n",
       "L 703 4441 \n",
       "L 1819 4666 \n",
       "L 2450 4666 \n",
       "L 2450 531 \n",
       "L 3481 531 \n",
       "L 3481 0 \n",
       "L 794 0 \n",
       "L 794 531 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "      </defs>\n",
       "      <use xlink:href=\"#DejaVuSans-78\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-31\" x=\"59.179688\"/>\n",
       "     </g>\n",
       "    </g>\n",
       "   </g>\n",
       "   <g id=\"matplotlib.axis_2\">\n",
       "    <g id=\"ytick_1\">\n",
       "     <g id=\"line2d_4\">\n",
       "      <defs>\n",
       "       <path id=\"m66f6a67b37\" d=\"M 0 0 \n",
       "L -3.5 0 \n",
       "\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </defs>\n",
       "      <g>\n",
       "       <use xlink:href=\"#m66f6a67b37\" x=\"42.620312\" y=\"143.1\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_5\">\n",
       "      <!-- −3 -->\n",
       "      <g transform=\"translate(20.878125 146.899219)scale(0.1 -0.1)\">\n",
       "       <defs>\n",
       "        <path id=\"DejaVuSans-33\" d=\"M 2597 2516 \n",
       "Q 3050 2419 3304 2112 \n",
       "Q 3559 1806 3559 1356 \n",
       "Q 3559 666 3084 287 \n",
       "Q 2609 -91 1734 -91 \n",
       "Q 1441 -91 1130 -33 \n",
       "Q 819 25 488 141 \n",
       "L 488 750 \n",
       "Q 750 597 1062 519 \n",
       "Q 1375 441 1716 441 \n",
       "Q 2309 441 2620 675 \n",
       "Q 2931 909 2931 1356 \n",
       "Q 2931 1769 2642 2001 \n",
       "Q 2353 2234 1838 2234 \n",
       "L 1294 2234 \n",
       "L 1294 2753 \n",
       "L 1863 2753 \n",
       "Q 2328 2753 2575 2939 \n",
       "Q 2822 3125 2822 3475 \n",
       "Q 2822 3834 2567 4026 \n",
       "Q 2313 4219 1838 4219 \n",
       "Q 1578 4219 1281 4162 \n",
       "Q 984 4106 628 3988 \n",
       "L 628 4550 \n",
       "Q 988 4650 1302 4700 \n",
       "Q 1616 4750 1894 4750 \n",
       "Q 2613 4750 3031 4423 \n",
       "Q 3450 4097 3450 3541 \n",
       "Q 3450 3153 3228 2886 \n",
       "Q 3006 2619 2597 2516 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       </defs>\n",
       "       <use xlink:href=\"#DejaVuSans-2212\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-33\" x=\"83.789062\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_2\">\n",
       "     <g id=\"line2d_5\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#m66f6a67b37\" x=\"42.620312\" y=\"108.253846\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_6\">\n",
       "      <!-- −2 -->\n",
       "      <g transform=\"translate(20.878125 112.053065)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#DejaVuSans-2212\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-32\" x=\"83.789062\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_3\">\n",
       "     <g id=\"line2d_6\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#m66f6a67b37\" x=\"42.620312\" y=\"73.407692\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_7\">\n",
       "      <!-- −1 -->\n",
       "      <g transform=\"translate(20.878125 77.206911)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#DejaVuSans-2212\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-31\" x=\"83.789062\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_4\">\n",
       "     <g id=\"line2d_7\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#m66f6a67b37\" x=\"42.620312\" y=\"38.561538\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_8\">\n",
       "      <!-- 0 -->\n",
       "      <g transform=\"translate(29.257812 42.360757)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#DejaVuSans-30\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"text_9\">\n",
       "     <!-- x2 -->\n",
       "     <g transform=\"translate(14.798437 81.290625)rotate(-90)scale(0.1 -0.1)\">\n",
       "      <use xlink:href=\"#DejaVuSans-78\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-32\" x=\"59.179688\"/>\n",
       "     </g>\n",
       "    </g>\n",
       "   </g>\n",
       "   <g id=\"line2d_8\">\n",
       "    <path d=\"M 57.878125 108.253846 \n",
       "L 88.39375 80.376923 \n",
       "L 112.80625 63.650769 \n",
       "L 132.33625 53.615076 \n",
       "L 147.96025 47.593661 \n",
       "L 160.45945 43.980812 \n",
       "L 170.45881 41.813102 \n",
       "L 178.458299 40.512476 \n",
       "L 184.857889 39.732101 \n",
       "L 189.977561 39.263876 \n",
       "L 194.073299 38.982941 \n",
       "L 197.349889 38.814379 \n",
       "L 199.971162 38.713243 \n",
       "L 202.068179 38.652561 \n",
       "L 203.745794 38.616152 \n",
       "L 205.087885 38.594306 \n",
       "L 206.161558 38.581199 \n",
       "L 207.020497 38.573334 \n",
       "L 207.707647 38.568616 \n",
       "L 208.257368 38.565785 \n",
       "L 208.697145 38.564086 \n",
       "\" clip-path=\"url(#p304360d856)\" style=\"fill: none; stroke: #ff7f0e; stroke-width: 1.5; stroke-linecap: square\"/>\n",
       "    <defs>\n",
       "     <path id=\"m961f3d55a7\" d=\"M 0 3 \n",
       "C 0.795609 3 1.55874 2.683901 2.12132 2.12132 \n",
       "C 2.683901 1.55874 3 0.795609 3 0 \n",
       "C 3 -0.795609 2.683901 -1.55874 2.12132 -2.12132 \n",
       "C 1.55874 -2.683901 0.795609 -3 0 -3 \n",
       "C -0.795609 -3 -1.55874 -2.683901 -2.12132 -2.12132 \n",
       "C -2.683901 -1.55874 -3 -0.795609 -3 0 \n",
       "C -3 0.795609 -2.683901 1.55874 -2.12132 2.12132 \n",
       "C -1.55874 2.683901 -0.795609 3 0 3 \n",
       "z\n",
       "\" style=\"stroke: #ff7f0e\"/>\n",
       "    </defs>\n",
       "    <g clip-path=\"url(#p304360d856)\">\n",
       "     <use xlink:href=\"#m961f3d55a7\" x=\"57.878125\" y=\"108.253846\" style=\"fill: #ff7f0e; stroke: #ff7f0e\"/>\n",
       "     <use xlink:href=\"#m961f3d55a7\" x=\"88.39375\" y=\"80.376923\" style=\"fill: #ff7f0e; stroke: #ff7f0e\"/>\n",
       "     <use xlink:href=\"#m961f3d55a7\" x=\"112.80625\" y=\"63.650769\" style=\"fill: #ff7f0e; stroke: #ff7f0e\"/>\n",
       "     <use xlink:href=\"#m961f3d55a7\" x=\"132.33625\" y=\"53.615076\" style=\"fill: #ff7f0e; stroke: #ff7f0e\"/>\n",
       "     <use xlink:href=\"#m961f3d55a7\" x=\"147.96025\" y=\"47.593661\" style=\"fill: #ff7f0e; stroke: #ff7f0e\"/>\n",
       "     <use xlink:href=\"#m961f3d55a7\" x=\"160.45945\" y=\"43.980812\" style=\"fill: #ff7f0e; stroke: #ff7f0e\"/>\n",
       "     <use xlink:href=\"#m961f3d55a7\" x=\"170.45881\" y=\"41.813102\" style=\"fill: #ff7f0e; stroke: #ff7f0e\"/>\n",
       "     <use xlink:href=\"#m961f3d55a7\" x=\"178.458299\" y=\"40.512476\" style=\"fill: #ff7f0e; stroke: #ff7f0e\"/>\n",
       "     <use xlink:href=\"#m961f3d55a7\" x=\"184.857889\" y=\"39.732101\" style=\"fill: #ff7f0e; stroke: #ff7f0e\"/>\n",
       "     <use xlink:href=\"#m961f3d55a7\" x=\"189.977561\" y=\"39.263876\" style=\"fill: #ff7f0e; stroke: #ff7f0e\"/>\n",
       "     <use xlink:href=\"#m961f3d55a7\" x=\"194.073299\" y=\"38.982941\" style=\"fill: #ff7f0e; stroke: #ff7f0e\"/>\n",
       "     <use xlink:href=\"#m961f3d55a7\" x=\"197.349889\" y=\"38.814379\" style=\"fill: #ff7f0e; stroke: #ff7f0e\"/>\n",
       "     <use xlink:href=\"#m961f3d55a7\" x=\"199.971162\" y=\"38.713243\" style=\"fill: #ff7f0e; stroke: #ff7f0e\"/>\n",
       "     <use xlink:href=\"#m961f3d55a7\" x=\"202.068179\" y=\"38.652561\" style=\"fill: #ff7f0e; stroke: #ff7f0e\"/>\n",
       "     <use xlink:href=\"#m961f3d55a7\" x=\"203.745794\" y=\"38.616152\" style=\"fill: #ff7f0e; stroke: #ff7f0e\"/>\n",
       "     <use xlink:href=\"#m961f3d55a7\" x=\"205.087885\" y=\"38.594306\" style=\"fill: #ff7f0e; stroke: #ff7f0e\"/>\n",
       "     <use xlink:href=\"#m961f3d55a7\" x=\"206.161558\" y=\"38.581199\" style=\"fill: #ff7f0e; stroke: #ff7f0e\"/>\n",
       "     <use xlink:href=\"#m961f3d55a7\" x=\"207.020497\" y=\"38.573334\" style=\"fill: #ff7f0e; stroke: #ff7f0e\"/>\n",
       "     <use xlink:href=\"#m961f3d55a7\" x=\"207.707647\" y=\"38.568616\" style=\"fill: #ff7f0e; stroke: #ff7f0e\"/>\n",
       "     <use xlink:href=\"#m961f3d55a7\" x=\"208.257368\" y=\"38.565785\" style=\"fill: #ff7f0e; stroke: #ff7f0e\"/>\n",
       "     <use xlink:href=\"#m961f3d55a7\" x=\"208.697145\" y=\"38.564086\" style=\"fill: #ff7f0e; stroke: #ff7f0e\"/>\n",
       "    </g>\n",
       "   </g>\n",
       "   <g id=\"PathCollection_1\"/>\n",
       "   <g id=\"PathCollection_2\">\n",
       "    <path d=\"M 133.389338 7.2 \n",
       "L 131.354961 10.684614 \n",
       "L 131.115628 11.149219 \n",
       "L 129.618631 14.16923 \n",
       "L 128.121641 17.653845 \n",
       "L 128.064069 17.812225 \n",
       "L 126.898922 21.138461 \n",
       "L 125.900228 24.623076 \n",
       "L 125.123465 28.107691 \n",
       "L 125.012502 28.804598 \n",
       "L 124.58421 31.592307 \n",
       "L 124.262994 35.076923 \n",
       "L 124.155921 38.561539 \n",
       "L 124.262994 42.046154 \n",
       "L 124.58421 45.530769 \n",
       "L 125.012502 48.318478 \n",
       "L 125.123465 49.015384 \n",
       "L 125.900228 52.500001 \n",
       "L 126.898922 55.984615 \n",
       "L 128.064069 59.310851 \n",
       "L 128.121641 59.469231 \n",
       "L 129.618631 62.953845 \n",
       "L 131.115628 65.973855 \n",
       "L 131.354961 66.438459 \n",
       "L 133.389338 69.923076 \n",
       "L 134.167188 71.115182 \n",
       "L 135.724107 73.407692 \n",
       "L 137.218755 75.398907 \n",
       "L 138.387435 76.892308 \n",
       "L 140.270314 79.089128 \n",
       "L 141.423126 80.37692 \n",
       "L 143.321874 82.328309 \n",
       "L 144.883135 83.861536 \n",
       "L 146.373441 85.216671 \n",
       "L 148.829573 87.346153 \n",
       "L 149.425 87.826789 \n",
       "L 152.476564 90.169895 \n",
       "L 153.383783 90.830769 \n",
       "L 155.528127 92.292059 \n",
       "L 158.57969 94.25918 \n",
       "L 158.672164 94.315385 \n",
       "L 161.631253 96.004895 \n",
       "L 164.682813 97.641606 \n",
       "L 164.998499 97.800001 \n",
       "L 167.734376 99.094286 \n",
       "L 170.785939 100.438353 \n",
       "L 172.861005 101.284618 \n",
       "L 173.837499 101.66133 \n",
       "L 176.889062 102.744387 \n",
       "L 179.940626 103.733262 \n",
       "L 182.992189 104.627962 \n",
       "L 183.530712 104.769234 \n",
       "L 186.04375 105.394675 \n",
       "L 189.095313 106.064794 \n",
       "L 192.146877 106.645562 \n",
       "L 195.198438 107.136982 \n",
       "L 198.250001 107.539054 \n",
       "L 201.301564 107.851774 \n",
       "L 204.353126 108.075148 \n",
       "L 207.404689 108.20917 \n",
       "L 210.456251 108.253846 \n",
       "L 213.507813 108.20917 \n",
       "L 216.559376 108.075148 \n",
       "L 219.610939 107.851774 \n",
       "L 222.662501 107.539054 \n",
       "L 225.714063 107.136982 \n",
       "L 228.765626 106.645562 \n",
       "L 231.817188 106.064794 \n",
       "L 234.868751 105.394675 \n",
       "L 237.38179 104.769234 \n",
       "L 237.920313 104.627962 \n",
       "\" clip-path=\"url(#p304360d856)\" style=\"fill: none; stroke: #1f77b4; stroke-width: 1.5\"/>\n",
       "   </g>\n",
       "   <g id=\"PathCollection_3\">\n",
       "    <path d=\"M 94.741 7.2 \n",
       "L 94.496869 7.814946 \n",
       "L 93.387217 10.684614 \n",
       "L 92.198295 14.16923 \n",
       "L 91.44531 16.715693 \n",
       "L 91.174923 17.653845 \n",
       "L 90.325119 21.138461 \n",
       "L 89.629825 24.623076 \n",
       "L 89.089044 28.107691 \n",
       "L 88.702769 31.592307 \n",
       "L 88.471007 35.076923 \n",
       "L 88.393757 38.561206 \n",
       "L 88.39375 38.561539 \n",
       "L 88.393757 38.561871 \n",
       "L 88.471007 42.046154 \n",
       "L 88.702769 45.530769 \n",
       "L 89.089044 49.015384 \n",
       "L 89.629825 52.500001 \n",
       "L 90.325119 55.984615 \n",
       "L 91.174923 59.469231 \n",
       "L 91.44531 60.407382 \n",
       "L 92.198295 62.953845 \n",
       "L 93.387217 66.438459 \n",
       "L 94.496869 69.30813 \n",
       "L 94.741 69.923076 \n",
       "L 96.287127 73.407692 \n",
       "L 97.548436 75.979666 \n",
       "L 98.008263 76.892308 \n",
       "L 99.931161 80.37692 \n",
       "L 100.599996 81.491996 \n",
       "L 102.061308 83.861536 \n",
       "L 103.651563 86.249145 \n",
       "L 104.403397 87.346153 \n",
       "L 106.703122 90.470284 \n",
       "L 106.976403 90.830769 \n",
       "L 109.754682 94.25917 \n",
       "L 109.801639 94.315385 \n",
       "L 112.806249 97.694405 \n",
       "L 112.903126 97.800001 \n",
       "L 115.857816 100.836595 \n",
       "L 116.308047 101.284618 \n",
       "L 118.909375 103.733262 \n",
       "L 120.04725 104.769234 \n",
       "L 121.960942 106.422193 \n",
       "L 124.155924 108.253846 \n",
       "L 125.012502 108.93377 \n",
       "L 128.064069 111.271016 \n",
       "L 128.697415 111.738466 \n",
       "L 131.115628 113.440255 \n",
       "L 133.748347 115.223078 \n",
       "L 134.167188 115.494103 \n",
       "L 137.218755 117.391282 \n",
       "L 139.426266 118.70769 \n",
       "L 140.270314 119.189606 \n",
       "L 143.321874 120.857774 \n",
       "L 145.876677 122.192311 \n",
       "L 146.373441 122.441211 \n",
       "L 149.425 123.899059 \n",
       "L 152.476564 125.285791 \n",
       "L 153.383788 125.676923 \n",
       "L 155.528127 126.565159 \n",
       "L 158.57969 127.760863 \n",
       "L 161.631253 128.888239 \n",
       "L 162.418734 129.161535 \n",
       "L 164.682813 129.917634 \n",
       "L 167.734376 130.870972 \n",
       "L 170.785939 131.758564 \n",
       "L 173.837499 132.580406 \n",
       "L 174.102859 132.646155 \n",
       "L 176.889062 133.311398 \n",
       "L 179.940626 133.976642 \n",
       "L 182.992189 134.57853 \n",
       "L 186.04375 135.117062 \n",
       "L 189.095313 135.592236 \n",
       "L 192.146877 136.004057 \n",
       "L 193.256511 136.130768 \n",
       "L 195.198438 136.344737 \n",
       "L 198.250001 136.619839 \n",
       "L 201.301564 136.833805 \n",
       "L 204.353126 136.986639 \n",
       "L 207.404689 137.078339 \n",
       "L 210.456251 137.108907 \n",
       "L 213.507813 137.078339 \n",
       "L 216.559376 136.986639 \n",
       "L 219.610939 136.833805 \n",
       "L 222.662501 136.619839 \n",
       "L 225.714063 136.344737 \n",
       "L 227.655991 136.130768 \n",
       "L 228.765626 136.004057 \n",
       "L 231.817188 135.592236 \n",
       "L 234.868751 135.117062 \n",
       "L 237.920313 134.57853 \n",
       "\" clip-path=\"url(#p304360d856)\" style=\"fill: none; stroke: #1f77b4; stroke-width: 1.5\"/>\n",
       "   </g>\n",
       "   <g id=\"PathCollection_4\">\n",
       "    <path d=\"M 66.101283 7.2 \n",
       "L 65.009145 10.684614 \n",
       "L 64.045489 14.16923 \n",
       "L 63.981244 14.437284 \n",
       "L 63.226219 17.653845 \n",
       "L 62.534115 21.138461 \n",
       "L 61.967845 24.623076 \n",
       "L 61.527415 28.107691 \n",
       "L 61.21282 31.592307 \n",
       "L 61.024065 35.076923 \n",
       "L 60.961145 38.561539 \n",
       "L 61.024065 42.046154 \n",
       "L 61.21282 45.530769 \n",
       "L 61.527415 49.015384 \n",
       "L 61.967845 52.500001 \n",
       "L 62.534115 55.984615 \n",
       "L 63.226219 59.469231 \n",
       "L 63.981244 62.685792 \n",
       "L 64.045489 62.953845 \n",
       "L 65.009145 66.438459 \n",
       "L 66.101283 69.923076 \n",
       "L 67.032818 72.582402 \n",
       "L 67.328125 73.407692 \n",
       "L 68.706252 76.892308 \n",
       "L 70.084378 80.073917 \n",
       "L 70.218509 80.37692 \n",
       "L 71.895191 83.861536 \n",
       "L 73.135938 86.249145 \n",
       "L 73.718821 87.346153 \n",
       "L 75.70748 90.830769 \n",
       "L 76.187512 91.617635 \n",
       "L 77.871125 94.315385 \n",
       "L 79.239071 96.374484 \n",
       "L 80.208385 97.800001 \n",
       "L 82.290631 100.687261 \n",
       "L 82.731817 101.284618 \n",
       "L 85.342191 104.627962 \n",
       "L 85.455214 104.769234 \n",
       "L 88.39375 108.253846 \n",
       "L 88.393757 108.253854 \n",
       "L 91.44531 111.610969 \n",
       "L 91.564212 111.738466 \n",
       "L 94.496869 114.736843 \n",
       "L 94.98513 115.223078 \n",
       "L 97.548436 117.662305 \n",
       "L 98.677098 118.70769 \n",
       "L 100.599996 120.412926 \n",
       "L 102.66303 122.192311 \n",
       "L 103.651563 123.010127 \n",
       "L 106.703122 125.463574 \n",
       "L 106.976403 125.676923 \n",
       "L 109.754682 127.760856 \n",
       "L 111.679511 129.161535 \n",
       "L 112.806249 129.950509 \n",
       "L 115.857816 132.021554 \n",
       "L 116.808306 132.646155 \n",
       "L 118.909375 133.97664 \n",
       "L 121.960942 135.845668 \n",
       "L 122.442758 136.130768 \n",
       "L 125.012502 137.597976 \n",
       "L 128.064069 139.279152 \n",
       "L 128.697416 139.615388 \n",
       "L 131.115628 140.855672 \n",
       "L 134.167188 142.361734 \n",
       "L 135.724104 143.1 \n",
       "\" clip-path=\"url(#p304360d856)\" style=\"fill: none; stroke: #1f77b4; stroke-width: 1.5\"/>\n",
       "   </g>\n",
       "   <g id=\"PathCollection_5\">\n",
       "    <path d=\"M 42.620312 71.115188 \n",
       "L 43.320213 73.407692 \n",
       "L 44.496043 76.892308 \n",
       "L 45.671872 80.073903 \n",
       "L 45.785952 80.37692 \n",
       "L 47.211915 83.861536 \n",
       "L 48.723432 87.281609 \n",
       "L 48.7525 87.346153 \n",
       "L 50.438126 90.830769 \n",
       "L 51.775006 93.41615 \n",
       "L 52.249024 94.315385 \n",
       "L 54.204398 97.800001 \n",
       "L 54.826565 98.845393 \n",
       "L 56.307021 101.284618 \n",
       "L 57.878125 103.733263 \n",
       "L 58.556253 104.769234 \n",
       "L 60.929685 108.209162 \n",
       "L 60.961151 108.253846 \n",
       "L 63.540821 111.738466 \n",
       "L 63.981244 112.305719 \n",
       "L 66.294014 115.223078 \n",
       "L 67.032818 116.113596 \n",
       "L 69.231252 118.70769 \n",
       "L 70.084378 119.671522 \n",
       "L 72.364666 122.192311 \n",
       "L 73.135938 123.010127 \n",
       "L 75.70748 125.676923 \n",
       "L 76.187512 126.155214 \n",
       "L 79.239071 129.127391 \n",
       "L 79.274952 129.161535 \n",
       "L 82.290631 131.922942 \n",
       "L 83.09947 132.646155 \n",
       "L 85.342191 134.578536 \n",
       "L 87.188189 136.130768 \n",
       "L 88.393757 137.108915 \n",
       "L 91.44531 139.52368 \n",
       "L 91.564212 139.615388 \n",
       "L 94.496869 141.800646 \n",
       "L 96.287127 143.1 \n",
       "\" clip-path=\"url(#p304360d856)\" style=\"fill: none; stroke: #1f77b4; stroke-width: 1.5\"/>\n",
       "   </g>\n",
       "   <g id=\"PathCollection_6\">\n",
       "    <path d=\"M 42.620312 115.494102 \n",
       "L 44.943982 118.70769 \n",
       "L 45.671872 119.671517 \n",
       "L 47.611182 122.192311 \n",
       "L 48.723432 123.579037 \n",
       "L 50.438126 125.676923 \n",
       "L 51.775006 127.248424 \n",
       "L 53.434101 129.161535 \n",
       "L 54.826565 130.706605 \n",
       "L 56.609163 132.646155 \n",
       "L 57.878125 133.97664 \n",
       "L 59.974143 136.130768 \n",
       "L 60.929685 137.078341 \n",
       "L 63.540815 139.615388 \n",
       "L 63.981244 140.028815 \n",
       "L 67.032818 142.834235 \n",
       "L 67.32812 143.1 \n",
       "\" clip-path=\"url(#p304360d856)\" style=\"fill: none; stroke: #1f77b4; stroke-width: 1.5\"/>\n",
       "   </g>\n",
       "   <g id=\"PathCollection_7\">\n",
       "    <path d=\"M 42.620312 142.361735 \n",
       "L 43.320214 143.1 \n",
       "\" clip-path=\"url(#p304360d856)\" style=\"fill: none; stroke: #1f77b4; stroke-width: 1.5\"/>\n",
       "   </g>\n",
       "   <g id=\"PathCollection_8\"/>\n",
       "   <g id=\"patch_3\">\n",
       "    <path d=\"M 42.620312 143.1 \n",
       "L 42.620312 7.2 \n",
       "\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_4\">\n",
       "    <path d=\"M 237.920313 143.1 \n",
       "L 237.920313 7.2 \n",
       "\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_5\">\n",
       "    <path d=\"M 42.620312 143.1 \n",
       "L 237.920313 143.1 \n",
       "\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_6\">\n",
       "    <path d=\"M 42.620312 7.2 \n",
       "L 237.920313 7.2 \n",
       "\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n",
       "   </g>\n",
       "  </g>\n",
       " </g>\n",
       " <defs>\n",
       "  <clipPath id=\"p304360d856\">\n",
       "   <rect x=\"42.620312\" y=\"7.2\" width=\"195.3\" height=\"135.9\"/>\n",
       "  </clipPath>\n",
       " </defs>\n",
       "</svg>\n"
      ],
      "text/plain": [
       "<Figure size 252x180 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "def f_2d(x1, x2):  # 目标函数\n",
    "    return x1 ** 2 + 2 * x2 ** 2\n",
    "\n",
    "def f_2d_grad(x1, x2):  # 目标函数的梯度\n",
    "    return (2 * x1, 4 * x2)\n",
    "\n",
    "def gd_2d(x1, x2, s1, s2, f_grad):\n",
    "    g1, g2 = f_grad(x1, x2)\n",
    "    return (x1 - eta * g1, x2 - eta * g2, 0, 0)\n",
    "\n",
    "eta = 0.1\n",
    "show_trace_2d(f_2d, train_2d(gd_2d, f_grad=f_2d_grad))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1e90e61f",
   "metadata": {
    "origin_pos": 25
   },
   "source": [
    "## 自适应方法\n",
    "\n",
    "正如我们在 :numref:`subsec_gd-learningrate`中所看到的，选择“恰到好处”的学习率$\\eta$是很棘手的。\n",
    "如果我们把它选得太小，就没有什么进展；如果太大，得到的解就会振荡，甚至可能发散。\n",
    "如果我们可以自动确定$\\eta$，或者完全不必选择学习率，会怎么样？\n",
    "除了考虑目标函数的值和梯度、还考虑它的曲率的二阶方法可以帮我们解决这个问题。\n",
    "虽然由于计算代价的原因，这些方法不能直接应用于深度学习，但它们为如何设计高级优化算法提供了有用的思维直觉，这些算法可以模拟下面概述的算法的许多理想特性。\n",
    "\n",
    "### 牛顿法\n",
    "\n",
    "回顾一些函数$f: \\mathbb{R}^d \\rightarrow \\mathbb{R}$的泰勒展开式，事实上我们可以把它写成\n",
    "\n",
    "$$f(\\mathbf{x} + \\boldsymbol{\\epsilon}) = f(\\mathbf{x}) + \\boldsymbol{\\epsilon}^\\top \\nabla f(\\mathbf{x}) + \\frac{1}{2} \\boldsymbol{\\epsilon}^\\top \\nabla^2 f(\\mathbf{x}) \\boldsymbol{\\epsilon} + \\mathcal{O}(\\|\\boldsymbol{\\epsilon}\\|^3).$$\n",
    ":eqlabel:`gd-hot-taylor`\n",
    "\n",
    "为了避免繁琐的符号，我们将$\\mathbf{H} \\stackrel{\\mathrm{def}}{=} \\nabla^2 f(\\mathbf{x})$定义为$f$的Hessian，是$d \\times d$矩阵。\n",
    "当$d$的值很小且问题很简单时，$\\mathbf{H}$很容易计算。\n",
    "但是对于深度神经网络而言，考虑到$\\mathbf{H}$可能非常大，\n",
    "$\\mathcal{O}(d^2)$个条目的存储代价会很高，\n",
    "此外通过反向传播进行计算可能雪上加霜。\n",
    "然而，我们姑且先忽略这些考量，看看会得到什么算法。\n",
    "\n",
    "毕竟，$f$的最小值满足$\\nabla f = 0$。\n",
    "遵循 :numref:`sec_calculus`中的微积分规则，\n",
    "通过取$\\boldsymbol{\\epsilon}$对 :eqref:`gd-hot-taylor`的导数，\n",
    "再忽略不重要的高阶项，我们便得到\n",
    "\n",
    "$$\\nabla f(\\mathbf{x}) + \\mathbf{H} \\boldsymbol{\\epsilon} = 0 \\text{ and hence }\n",
    "\\boldsymbol{\\epsilon} = -\\mathbf{H}^{-1} \\nabla f(\\mathbf{x}).$$\n",
    "\n",
    "也就是说，作为优化问题的一部分，我们需要将Hessian矩阵$\\mathbf{H}$求逆。\n",
    "\n",
    "举一个简单的例子，对于$f(x) = \\frac{1}{2} x^2$，我们有$\\nabla f(x) = x$和$\\mathbf{H} = 1$。\n",
    "因此，对于任何$x$，我们可以获得$\\epsilon = -x$。\n",
    "换言之，单单一步就足以完美地收敛，而无须任何调整。\n",
    "我们在这里比较幸运：泰勒展开式是确切的，因为$f(x+\\epsilon)= \\frac{1}{2} x^2 + \\epsilon x + \\frac{1}{2} \\epsilon^2$。\n",
    "\n",
    "让我们看看其他问题。\n",
    "给定一个凸双曲余弦函数$c$，其中$c$为某些常数，\n",
    "我们可以看到经过几次迭代后，得到了$x=0$处的全局最小值。\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "de14c854",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-08-18T07:01:27.887060Z",
     "iopub.status.busy": "2023-08-18T07:01:27.886284Z",
     "iopub.status.idle": "2023-08-18T07:01:28.178108Z",
     "shell.execute_reply": "2023-08-18T07:01:28.176813Z"
    },
    "origin_pos": 26,
    "tab": [
     "pytorch"
    ]
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "epoch 10, x: tensor(0.)\n"
     ]
    },
    {
     "data": {
      "image/svg+xml": [
       "<?xml version=\"1.0\" encoding=\"utf-8\" standalone=\"no\"?>\n",
       "<!DOCTYPE svg PUBLIC \"-//W3C//DTD SVG 1.1//EN\"\n",
       "  \"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd\">\n",
       "<svg xmlns:xlink=\"http://www.w3.org/1999/xlink\" width=\"243.103125pt\" height=\"180.65625pt\" viewBox=\"0 0 243.103125 180.65625\" xmlns=\"http://www.w3.org/2000/svg\" version=\"1.1\">\n",
       " <metadata>\n",
       "  <rdf:RDF xmlns:dc=\"http://purl.org/dc/elements/1.1/\" xmlns:cc=\"http://creativecommons.org/ns#\" xmlns:rdf=\"http://www.w3.org/1999/02/22-rdf-syntax-ns#\">\n",
       "   <cc:Work>\n",
       "    <dc:type rdf:resource=\"http://purl.org/dc/dcmitype/StillImage\"/>\n",
       "    <dc:date>2023-08-18T07:01:28.130732</dc:date>\n",
       "    <dc:format>image/svg+xml</dc:format>\n",
       "    <dc:creator>\n",
       "     <cc:Agent>\n",
       "      <dc:title>Matplotlib v3.5.1, https://matplotlib.org/</dc:title>\n",
       "     </cc:Agent>\n",
       "    </dc:creator>\n",
       "   </cc:Work>\n",
       "  </rdf:RDF>\n",
       " </metadata>\n",
       " <defs>\n",
       "  <style type=\"text/css\">*{stroke-linejoin: round; stroke-linecap: butt}</style>\n",
       " </defs>\n",
       " <g id=\"figure_1\">\n",
       "  <g id=\"patch_1\">\n",
       "   <path d=\"M 0 180.65625 \n",
       "L 243.103125 180.65625 \n",
       "L 243.103125 0 \n",
       "L 0 0 \n",
       "L 0 180.65625 \n",
       "z\n",
       "\" style=\"fill: none\"/>\n",
       "  </g>\n",
       "  <g id=\"axes_1\">\n",
       "   <g id=\"patch_2\">\n",
       "    <path d=\"M 40.603125 143.1 \n",
       "L 235.903125 143.1 \n",
       "L 235.903125 7.2 \n",
       "L 40.603125 7.2 \n",
       "z\n",
       "\" style=\"fill: #ffffff\"/>\n",
       "   </g>\n",
       "   <g id=\"matplotlib.axis_1\">\n",
       "    <g id=\"xtick_1\">\n",
       "     <g id=\"line2d_1\">\n",
       "      <path d=\"M 49.480398 143.1 \n",
       "L 49.480398 7.2 \n",
       "\" clip-path=\"url(#p20691e33d4)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_2\">\n",
       "      <defs>\n",
       "       <path id=\"mb45305dd2c\" d=\"M 0 0 \n",
       "L 0 3.5 \n",
       "\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </defs>\n",
       "      <g>\n",
       "       <use xlink:href=\"#mb45305dd2c\" x=\"49.480398\" y=\"143.1\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_1\">\n",
       "      <!-- −10 -->\n",
       "      <g transform=\"translate(38.928054 157.698438)scale(0.1 -0.1)\">\n",
       "       <defs>\n",
       "        <path id=\"DejaVuSans-2212\" d=\"M 678 2272 \n",
       "L 4684 2272 \n",
       "L 4684 1741 \n",
       "L 678 1741 \n",
       "L 678 2272 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "        <path id=\"DejaVuSans-31\" d=\"M 794 531 \n",
       "L 1825 531 \n",
       "L 1825 4091 \n",
       "L 703 3866 \n",
       "L 703 4441 \n",
       "L 1819 4666 \n",
       "L 2450 4666 \n",
       "L 2450 531 \n",
       "L 3481 531 \n",
       "L 3481 0 \n",
       "L 794 0 \n",
       "L 794 531 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "        <path id=\"DejaVuSans-30\" d=\"M 2034 4250 \n",
       "Q 1547 4250 1301 3770 \n",
       "Q 1056 3291 1056 2328 \n",
       "Q 1056 1369 1301 889 \n",
       "Q 1547 409 2034 409 \n",
       "Q 2525 409 2770 889 \n",
       "Q 3016 1369 3016 2328 \n",
       "Q 3016 3291 2770 3770 \n",
       "Q 2525 4250 2034 4250 \n",
       "z\n",
       "M 2034 4750 \n",
       "Q 2819 4750 3233 4129 \n",
       "Q 3647 3509 3647 2328 \n",
       "Q 3647 1150 3233 529 \n",
       "Q 2819 -91 2034 -91 \n",
       "Q 1250 -91 836 529 \n",
       "Q 422 1150 422 2328 \n",
       "Q 422 3509 836 4129 \n",
       "Q 1250 4750 2034 4750 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       </defs>\n",
       "       <use xlink:href=\"#DejaVuSans-2212\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-31\" x=\"83.789062\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"147.412109\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_2\">\n",
       "     <g id=\"line2d_3\">\n",
       "      <path d=\"M 93.866761 143.1 \n",
       "L 93.866761 7.2 \n",
       "\" clip-path=\"url(#p20691e33d4)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_4\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#mb45305dd2c\" x=\"93.866761\" y=\"143.1\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_2\">\n",
       "      <!-- −5 -->\n",
       "      <g transform=\"translate(86.495668 157.698438)scale(0.1 -0.1)\">\n",
       "       <defs>\n",
       "        <path id=\"DejaVuSans-35\" d=\"M 691 4666 \n",
       "L 3169 4666 \n",
       "L 3169 4134 \n",
       "L 1269 4134 \n",
       "L 1269 2991 \n",
       "Q 1406 3038 1543 3061 \n",
       "Q 1681 3084 1819 3084 \n",
       "Q 2600 3084 3056 2656 \n",
       "Q 3513 2228 3513 1497 \n",
       "Q 3513 744 3044 326 \n",
       "Q 2575 -91 1722 -91 \n",
       "Q 1428 -91 1123 -41 \n",
       "Q 819 9 494 109 \n",
       "L 494 744 \n",
       "Q 775 591 1075 516 \n",
       "Q 1375 441 1709 441 \n",
       "Q 2250 441 2565 725 \n",
       "Q 2881 1009 2881 1497 \n",
       "Q 2881 1984 2565 2268 \n",
       "Q 2250 2553 1709 2553 \n",
       "Q 1456 2553 1204 2497 \n",
       "Q 953 2441 691 2322 \n",
       "L 691 4666 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       </defs>\n",
       "       <use xlink:href=\"#DejaVuSans-2212\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-35\" x=\"83.789062\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_3\">\n",
       "     <g id=\"line2d_5\">\n",
       "      <path d=\"M 138.253125 143.1 \n",
       "L 138.253125 7.2 \n",
       "\" clip-path=\"url(#p20691e33d4)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_6\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#mb45305dd2c\" x=\"138.253125\" y=\"143.1\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_3\">\n",
       "      <!-- 0 -->\n",
       "      <g transform=\"translate(135.071875 157.698438)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#DejaVuSans-30\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_4\">\n",
       "     <g id=\"line2d_7\">\n",
       "      <path d=\"M 182.639489 143.1 \n",
       "L 182.639489 7.2 \n",
       "\" clip-path=\"url(#p20691e33d4)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_8\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#mb45305dd2c\" x=\"182.639489\" y=\"143.1\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_4\">\n",
       "      <!-- 5 -->\n",
       "      <g transform=\"translate(179.458239 157.698438)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#DejaVuSans-35\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_5\">\n",
       "     <g id=\"line2d_9\">\n",
       "      <path d=\"M 227.025852 143.1 \n",
       "L 227.025852 7.2 \n",
       "\" clip-path=\"url(#p20691e33d4)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_10\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#mb45305dd2c\" x=\"227.025852\" y=\"143.1\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_5\">\n",
       "      <!-- 10 -->\n",
       "      <g transform=\"translate(220.663352 157.698438)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#DejaVuSans-31\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"63.623047\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"text_6\">\n",
       "     <!-- x -->\n",
       "     <g transform=\"translate(135.29375 171.376563)scale(0.1 -0.1)\">\n",
       "      <defs>\n",
       "       <path id=\"DejaVuSans-78\" d=\"M 3513 3500 \n",
       "L 2247 1797 \n",
       "L 3578 0 \n",
       "L 2900 0 \n",
       "L 1881 1375 \n",
       "L 863 0 \n",
       "L 184 0 \n",
       "L 1544 1831 \n",
       "L 300 3500 \n",
       "L 978 3500 \n",
       "L 1906 2253 \n",
       "L 2834 3500 \n",
       "L 3513 3500 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "      </defs>\n",
       "      <use xlink:href=\"#DejaVuSans-78\"/>\n",
       "     </g>\n",
       "    </g>\n",
       "   </g>\n",
       "   <g id=\"matplotlib.axis_2\">\n",
       "    <g id=\"ytick_1\">\n",
       "     <g id=\"line2d_11\">\n",
       "      <path d=\"M 40.603125 138.610277 \n",
       "L 235.903125 138.610277 \n",
       "\" clip-path=\"url(#p20691e33d4)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_12\">\n",
       "      <defs>\n",
       "       <path id=\"m35bd69b896\" d=\"M 0 0 \n",
       "L -3.5 0 \n",
       "\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </defs>\n",
       "      <g>\n",
       "       <use xlink:href=\"#m35bd69b896\" x=\"40.603125\" y=\"138.610277\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_7\">\n",
       "      <!-- 0 -->\n",
       "      <g transform=\"translate(27.240625 142.409496)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#DejaVuSans-30\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_2\">\n",
       "     <g id=\"line2d_13\">\n",
       "      <path d=\"M 40.603125 104.859278 \n",
       "L 235.903125 104.859278 \n",
       "\" clip-path=\"url(#p20691e33d4)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_14\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#m35bd69b896\" x=\"40.603125\" y=\"104.859278\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_8\">\n",
       "      <!-- 20 -->\n",
       "      <g transform=\"translate(20.878125 108.658497)scale(0.1 -0.1)\">\n",
       "       <defs>\n",
       "        <path id=\"DejaVuSans-32\" d=\"M 1228 531 \n",
       "L 3431 531 \n",
       "L 3431 0 \n",
       "L 469 0 \n",
       "L 469 531 \n",
       "Q 828 903 1448 1529 \n",
       "Q 2069 2156 2228 2338 \n",
       "Q 2531 2678 2651 2914 \n",
       "Q 2772 3150 2772 3378 \n",
       "Q 2772 3750 2511 3984 \n",
       "Q 2250 4219 1831 4219 \n",
       "Q 1534 4219 1204 4116 \n",
       "Q 875 4013 500 3803 \n",
       "L 500 4441 \n",
       "Q 881 4594 1212 4672 \n",
       "Q 1544 4750 1819 4750 \n",
       "Q 2544 4750 2975 4387 \n",
       "Q 3406 4025 3406 3419 \n",
       "Q 3406 3131 3298 2873 \n",
       "Q 3191 2616 2906 2266 \n",
       "Q 2828 2175 2409 1742 \n",
       "Q 1991 1309 1228 531 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       </defs>\n",
       "       <use xlink:href=\"#DejaVuSans-32\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"63.623047\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_3\">\n",
       "     <g id=\"line2d_15\">\n",
       "      <path d=\"M 40.603125 71.108278 \n",
       "L 235.903125 71.108278 \n",
       "\" clip-path=\"url(#p20691e33d4)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_16\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#m35bd69b896\" x=\"40.603125\" y=\"71.108278\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_9\">\n",
       "      <!-- 40 -->\n",
       "      <g transform=\"translate(20.878125 74.907497)scale(0.1 -0.1)\">\n",
       "       <defs>\n",
       "        <path id=\"DejaVuSans-34\" d=\"M 2419 4116 \n",
       "L 825 1625 \n",
       "L 2419 1625 \n",
       "L 2419 4116 \n",
       "z\n",
       "M 2253 4666 \n",
       "L 3047 4666 \n",
       "L 3047 1625 \n",
       "L 3713 1625 \n",
       "L 3713 1100 \n",
       "L 3047 1100 \n",
       "L 3047 0 \n",
       "L 2419 0 \n",
       "L 2419 1100 \n",
       "L 313 1100 \n",
       "L 313 1709 \n",
       "L 2253 4666 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       </defs>\n",
       "       <use xlink:href=\"#DejaVuSans-34\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"63.623047\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_4\">\n",
       "     <g id=\"line2d_17\">\n",
       "      <path d=\"M 40.603125 37.357279 \n",
       "L 235.903125 37.357279 \n",
       "\" clip-path=\"url(#p20691e33d4)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_18\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#m35bd69b896\" x=\"40.603125\" y=\"37.357279\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_10\">\n",
       "      <!-- 60 -->\n",
       "      <g transform=\"translate(20.878125 41.156498)scale(0.1 -0.1)\">\n",
       "       <defs>\n",
       "        <path id=\"DejaVuSans-36\" d=\"M 2113 2584 \n",
       "Q 1688 2584 1439 2293 \n",
       "Q 1191 2003 1191 1497 \n",
       "Q 1191 994 1439 701 \n",
       "Q 1688 409 2113 409 \n",
       "Q 2538 409 2786 701 \n",
       "Q 3034 994 3034 1497 \n",
       "Q 3034 2003 2786 2293 \n",
       "Q 2538 2584 2113 2584 \n",
       "z\n",
       "M 3366 4563 \n",
       "L 3366 3988 \n",
       "Q 3128 4100 2886 4159 \n",
       "Q 2644 4219 2406 4219 \n",
       "Q 1781 4219 1451 3797 \n",
       "Q 1122 3375 1075 2522 \n",
       "Q 1259 2794 1537 2939 \n",
       "Q 1816 3084 2150 3084 \n",
       "Q 2853 3084 3261 2657 \n",
       "Q 3669 2231 3669 1497 \n",
       "Q 3669 778 3244 343 \n",
       "Q 2819 -91 2113 -91 \n",
       "Q 1303 -91 875 529 \n",
       "Q 447 1150 447 2328 \n",
       "Q 447 3434 972 4092 \n",
       "Q 1497 4750 2381 4750 \n",
       "Q 2619 4750 2861 4703 \n",
       "Q 3103 4656 3366 4563 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       </defs>\n",
       "       <use xlink:href=\"#DejaVuSans-36\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"63.623047\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"text_11\">\n",
       "     <!-- f(x) -->\n",
       "     <g transform=\"translate(14.798437 83.771094)rotate(-90)scale(0.1 -0.1)\">\n",
       "      <defs>\n",
       "       <path id=\"DejaVuSans-66\" d=\"M 2375 4863 \n",
       "L 2375 4384 \n",
       "L 1825 4384 \n",
       "Q 1516 4384 1395 4259 \n",
       "Q 1275 4134 1275 3809 \n",
       "L 1275 3500 \n",
       "L 2222 3500 \n",
       "L 2222 3053 \n",
       "L 1275 3053 \n",
       "L 1275 0 \n",
       "L 697 0 \n",
       "L 697 3053 \n",
       "L 147 3053 \n",
       "L 147 3500 \n",
       "L 697 3500 \n",
       "L 697 3744 \n",
       "Q 697 4328 969 4595 \n",
       "Q 1241 4863 1831 4863 \n",
       "L 2375 4863 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-28\" d=\"M 1984 4856 \n",
       "Q 1566 4138 1362 3434 \n",
       "Q 1159 2731 1159 2009 \n",
       "Q 1159 1288 1364 580 \n",
       "Q 1569 -128 1984 -844 \n",
       "L 1484 -844 \n",
       "Q 1016 -109 783 600 \n",
       "Q 550 1309 550 2009 \n",
       "Q 550 2706 781 3412 \n",
       "Q 1013 4119 1484 4856 \n",
       "L 1984 4856 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-29\" d=\"M 513 4856 \n",
       "L 1013 4856 \n",
       "Q 1481 4119 1714 3412 \n",
       "Q 1947 2706 1947 2009 \n",
       "Q 1947 1309 1714 600 \n",
       "Q 1481 -109 1013 -844 \n",
       "L 513 -844 \n",
       "Q 928 -128 1133 580 \n",
       "Q 1338 1288 1338 2009 \n",
       "Q 1338 2731 1133 3434 \n",
       "Q 928 4138 513 4856 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "      </defs>\n",
       "      <use xlink:href=\"#DejaVuSans-66\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-28\" x=\"35.205078\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-78\" x=\"74.21875\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-29\" x=\"133.398438\"/>\n",
       "     </g>\n",
       "    </g>\n",
       "   </g>\n",
       "   <g id=\"line2d_19\">\n",
       "    <path d=\"M 49.480398 13.377273 \n",
       "L 51.522166 26.980492 \n",
       "L 53.563944 39.105811 \n",
       "L 55.605721 49.913743 \n",
       "L 57.647489 59.547321 \n",
       "L 59.689258 68.134144 \n",
       "L 61.731035 75.787923 \n",
       "L 63.772812 82.609959 \n",
       "L 65.814581 88.690547 \n",
       "L 67.945124 94.332011 \n",
       "L 70.075672 99.335114 \n",
       "L 72.206215 103.771963 \n",
       "L 74.336763 107.706548 \n",
       "L 76.556081 111.332057 \n",
       "L 78.775399 114.530783 \n",
       "L 80.994718 117.352778 \n",
       "L 83.302811 119.93542 \n",
       "L 85.69967 122.283408 \n",
       "L 88.185308 124.404097 \n",
       "L 90.759717 126.306978 \n",
       "L 93.4229 128.003223 \n",
       "L 96.263625 129.549831 \n",
       "L 99.281899 130.939216 \n",
       "L 102.566489 132.19986 \n",
       "L 106.117397 133.316347 \n",
       "L 109.934624 134.280667 \n",
       "L 114.195716 135.121514 \n",
       "L 118.90067 135.816968 \n",
       "L 124.138261 136.360754 \n",
       "L 129.908489 136.732881 \n",
       "L 136.12258 136.910562 \n",
       "L 142.514216 136.873892 \n",
       "L 148.728306 136.620389 \n",
       "L 154.498534 136.165615 \n",
       "L 159.647353 135.541931 \n",
       "L 164.263535 134.763827 \n",
       "L 168.435851 133.837354 \n",
       "L 172.164306 132.787244 \n",
       "L 175.626444 131.582519 \n",
       "L 178.822263 130.233733 \n",
       "L 181.751762 128.759507 \n",
       "L 184.503716 127.13061 \n",
       "L 187.1669 125.291583 \n",
       "L 189.652538 123.305207 \n",
       "L 192.049397 121.105704 \n",
       "L 194.35749 118.686198 \n",
       "L 196.576808 116.042311 \n",
       "L 198.796127 113.045339 \n",
       "L 200.92667 109.792602 \n",
       "L 203.057218 106.124383 \n",
       "L 205.09899 102.17027 \n",
       "L 207.140763 97.733708 \n",
       "L 209.182536 92.755956 \n",
       "L 211.224309 87.171112 \n",
       "L 213.266078 80.905247 \n",
       "L 215.21908 74.198133 \n",
       "L 217.172083 66.71084 \n",
       "L 219.125076 58.352722 \n",
       "L 221.078079 49.022466 \n",
       "L 223.031081 38.607112 \n",
       "L 224.984084 26.980492 \n",
       "L 226.937078 14.001838 \n",
       "L 226.937078 14.001838 \n",
       "\" clip-path=\"url(#p20691e33d4)\" style=\"fill: none; stroke: #1f77b4; stroke-width: 1.5; stroke-linecap: square\"/>\n",
       "   </g>\n",
       "   <g id=\"line2d_20\">\n",
       "    <path d=\"M 227.025852 13.377273 \n",
       "L 209.272924 92.522075 \n",
       "L 191.530286 121.607711 \n",
       "L 173.86341 132.226396 \n",
       "L 156.74043 135.92216 \n",
       "L 142.920474 136.86408 \n",
       "L 138.357749 136.922698 \n",
       "L 138.253126 136.922727 \n",
       "L 138.253125 136.922727 \n",
       "L 138.253125 136.922727 \n",
       "L 138.253125 136.922727 \n",
       "\" clip-path=\"url(#p20691e33d4)\" style=\"fill: none; stroke: #ff7f0e; stroke-width: 1.5; stroke-linecap: square\"/>\n",
       "    <defs>\n",
       "     <path id=\"m4983b9dc7d\" d=\"M 0 3 \n",
       "C 0.795609 3 1.55874 2.683901 2.12132 2.12132 \n",
       "C 2.683901 1.55874 3 0.795609 3 0 \n",
       "C 3 -0.795609 2.683901 -1.55874 2.12132 -2.12132 \n",
       "C 1.55874 -2.683901 0.795609 -3 0 -3 \n",
       "C -0.795609 -3 -1.55874 -2.683901 -2.12132 -2.12132 \n",
       "C -2.683901 -1.55874 -3 -0.795609 -3 0 \n",
       "C -3 0.795609 -2.683901 1.55874 -2.12132 2.12132 \n",
       "C -1.55874 2.683901 -0.795609 3 0 3 \n",
       "z\n",
       "\" style=\"stroke: #ff7f0e\"/>\n",
       "    </defs>\n",
       "    <g clip-path=\"url(#p20691e33d4)\">\n",
       "     <use xlink:href=\"#m4983b9dc7d\" x=\"227.025852\" y=\"13.377273\" style=\"fill: #ff7f0e; stroke: #ff7f0e\"/>\n",
       "     <use xlink:href=\"#m4983b9dc7d\" x=\"209.272924\" y=\"92.522075\" style=\"fill: #ff7f0e; stroke: #ff7f0e\"/>\n",
       "     <use xlink:href=\"#m4983b9dc7d\" x=\"191.530286\" y=\"121.607711\" style=\"fill: #ff7f0e; stroke: #ff7f0e\"/>\n",
       "     <use xlink:href=\"#m4983b9dc7d\" x=\"173.86341\" y=\"132.226396\" style=\"fill: #ff7f0e; stroke: #ff7f0e\"/>\n",
       "     <use xlink:href=\"#m4983b9dc7d\" x=\"156.74043\" y=\"135.92216\" style=\"fill: #ff7f0e; stroke: #ff7f0e\"/>\n",
       "     <use xlink:href=\"#m4983b9dc7d\" x=\"142.920474\" y=\"136.86408\" style=\"fill: #ff7f0e; stroke: #ff7f0e\"/>\n",
       "     <use xlink:href=\"#m4983b9dc7d\" x=\"138.357749\" y=\"136.922698\" style=\"fill: #ff7f0e; stroke: #ff7f0e\"/>\n",
       "     <use xlink:href=\"#m4983b9dc7d\" x=\"138.253126\" y=\"136.922727\" style=\"fill: #ff7f0e; stroke: #ff7f0e\"/>\n",
       "     <use xlink:href=\"#m4983b9dc7d\" x=\"138.253125\" y=\"136.922727\" style=\"fill: #ff7f0e; stroke: #ff7f0e\"/>\n",
       "     <use xlink:href=\"#m4983b9dc7d\" x=\"138.253125\" y=\"136.922727\" style=\"fill: #ff7f0e; stroke: #ff7f0e\"/>\n",
       "     <use xlink:href=\"#m4983b9dc7d\" x=\"138.253125\" y=\"136.922727\" style=\"fill: #ff7f0e; stroke: #ff7f0e\"/>\n",
       "    </g>\n",
       "   </g>\n",
       "   <g id=\"patch_3\">\n",
       "    <path d=\"M 40.603125 143.1 \n",
       "L 40.603125 7.2 \n",
       "\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_4\">\n",
       "    <path d=\"M 235.903125 143.1 \n",
       "L 235.903125 7.2 \n",
       "\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_5\">\n",
       "    <path d=\"M 40.603125 143.1 \n",
       "L 235.903125 143.1 \n",
       "\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_6\">\n",
       "    <path d=\"M 40.603125 7.2 \n",
       "L 235.903125 7.2 \n",
       "\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n",
       "   </g>\n",
       "  </g>\n",
       " </g>\n",
       " <defs>\n",
       "  <clipPath id=\"p20691e33d4\">\n",
       "   <rect x=\"40.603125\" y=\"7.2\" width=\"195.3\" height=\"135.9\"/>\n",
       "  </clipPath>\n",
       " </defs>\n",
       "</svg>\n"
      ],
      "text/plain": [
       "<Figure size 252x180 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "c = torch.tensor(0.5)\n",
    "\n",
    "def f(x):  # O目标函数\n",
    "    return torch.cosh(c * x)\n",
    "\n",
    "def f_grad(x):  # 目标函数的梯度\n",
    "    return c * torch.sinh(c * x)\n",
    "\n",
    "def f_hess(x):  # 目标函数的Hessian\n",
    "    return c**2 * torch.cosh(c * x)\n",
    "\n",
    "def newton(eta=1):\n",
    "    x = 10.0\n",
    "    results = [x]\n",
    "    for i in range(10):\n",
    "        x -= eta * f_grad(x) / f_hess(x)\n",
    "        results.append(float(x))\n",
    "    print('epoch 10, x:', x)\n",
    "    return results\n",
    "\n",
    "show_trace(newton(), f)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "61c98547",
   "metadata": {
    "origin_pos": 27
   },
   "source": [
    "现在让我们考虑一个非凸函数，比如$f(x) = x \\cos(c x)$，$c$为某些常数。\n",
    "请注意在牛顿法中，我们最终将除以Hessian。\n",
    "这意味着如果二阶导数是负的，$f$的值可能会趋于增加。\n",
    "这是这个算法的致命缺陷！\n",
    "让我们看看实践中会发生什么。\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "302d97de",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-08-18T07:01:28.183563Z",
     "iopub.status.busy": "2023-08-18T07:01:28.182810Z",
     "iopub.status.idle": "2023-08-18T07:01:28.660824Z",
     "shell.execute_reply": "2023-08-18T07:01:28.659664Z"
    },
    "origin_pos": 28,
    "tab": [
     "pytorch"
    ]
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "epoch 10, x: tensor(26.8341)\n"
     ]
    },
    {
     "data": {
      "image/svg+xml": [
       "<?xml version=\"1.0\" encoding=\"utf-8\" standalone=\"no\"?>\n",
       "<!DOCTYPE svg PUBLIC \"-//W3C//DTD SVG 1.1//EN\"\n",
       "  \"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd\">\n",
       "<svg xmlns:xlink=\"http://www.w3.org/1999/xlink\" width=\"251.482813pt\" height=\"180.65625pt\" viewBox=\"0 0 251.482813 180.65625\" xmlns=\"http://www.w3.org/2000/svg\" version=\"1.1\">\n",
       " <metadata>\n",
       "  <rdf:RDF xmlns:dc=\"http://purl.org/dc/elements/1.1/\" xmlns:cc=\"http://creativecommons.org/ns#\" xmlns:rdf=\"http://www.w3.org/1999/02/22-rdf-syntax-ns#\">\n",
       "   <cc:Work>\n",
       "    <dc:type rdf:resource=\"http://purl.org/dc/dcmitype/StillImage\"/>\n",
       "    <dc:date>2023-08-18T07:01:28.616405</dc:date>\n",
       "    <dc:format>image/svg+xml</dc:format>\n",
       "    <dc:creator>\n",
       "     <cc:Agent>\n",
       "      <dc:title>Matplotlib v3.5.1, https://matplotlib.org/</dc:title>\n",
       "     </cc:Agent>\n",
       "    </dc:creator>\n",
       "   </cc:Work>\n",
       "  </rdf:RDF>\n",
       " </metadata>\n",
       " <defs>\n",
       "  <style type=\"text/css\">*{stroke-linejoin: round; stroke-linecap: butt}</style>\n",
       " </defs>\n",
       " <g id=\"figure_1\">\n",
       "  <g id=\"patch_1\">\n",
       "   <path d=\"M -0 180.65625 \n",
       "L 251.482813 180.65625 \n",
       "L 251.482813 0 \n",
       "L -0 0 \n",
       "L -0 180.65625 \n",
       "z\n",
       "\" style=\"fill: none\"/>\n",
       "  </g>\n",
       "  <g id=\"axes_1\">\n",
       "   <g id=\"patch_2\">\n",
       "    <path d=\"M 48.982813 143.1 \n",
       "L 244.282813 143.1 \n",
       "L 244.282813 7.2 \n",
       "L 48.982813 7.2 \n",
       "z\n",
       "\" style=\"fill: #ffffff\"/>\n",
       "   </g>\n",
       "   <g id=\"matplotlib.axis_1\">\n",
       "    <g id=\"xtick_1\">\n",
       "     <g id=\"line2d_1\">\n",
       "      <path d=\"M 82.72875 143.1 \n",
       "L 82.72875 7.2 \n",
       "\" clip-path=\"url(#pc9dd1287a9)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_2\">\n",
       "      <defs>\n",
       "       <path id=\"mf2d922344a\" d=\"M 0 0 \n",
       "L 0 3.5 \n",
       "\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </defs>\n",
       "      <g>\n",
       "       <use xlink:href=\"#mf2d922344a\" x=\"82.72875\" y=\"143.1\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_1\">\n",
       "      <!-- −20 -->\n",
       "      <g transform=\"translate(72.176406 157.698438)scale(0.1 -0.1)\">\n",
       "       <defs>\n",
       "        <path id=\"DejaVuSans-2212\" d=\"M 678 2272 \n",
       "L 4684 2272 \n",
       "L 4684 1741 \n",
       "L 678 1741 \n",
       "L 678 2272 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "        <path id=\"DejaVuSans-32\" d=\"M 1228 531 \n",
       "L 3431 531 \n",
       "L 3431 0 \n",
       "L 469 0 \n",
       "L 469 531 \n",
       "Q 828 903 1448 1529 \n",
       "Q 2069 2156 2228 2338 \n",
       "Q 2531 2678 2651 2914 \n",
       "Q 2772 3150 2772 3378 \n",
       "Q 2772 3750 2511 3984 \n",
       "Q 2250 4219 1831 4219 \n",
       "Q 1534 4219 1204 4116 \n",
       "Q 875 4013 500 3803 \n",
       "L 500 4441 \n",
       "Q 881 4594 1212 4672 \n",
       "Q 1544 4750 1819 4750 \n",
       "Q 2544 4750 2975 4387 \n",
       "Q 3406 4025 3406 3419 \n",
       "Q 3406 3131 3298 2873 \n",
       "Q 3191 2616 2906 2266 \n",
       "Q 2828 2175 2409 1742 \n",
       "Q 1991 1309 1228 531 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "        <path id=\"DejaVuSans-30\" d=\"M 2034 4250 \n",
       "Q 1547 4250 1301 3770 \n",
       "Q 1056 3291 1056 2328 \n",
       "Q 1056 1369 1301 889 \n",
       "Q 1547 409 2034 409 \n",
       "Q 2525 409 2770 889 \n",
       "Q 3016 1369 3016 2328 \n",
       "Q 3016 3291 2770 3770 \n",
       "Q 2525 4250 2034 4250 \n",
       "z\n",
       "M 2034 4750 \n",
       "Q 2819 4750 3233 4129 \n",
       "Q 3647 3509 3647 2328 \n",
       "Q 3647 1150 3233 529 \n",
       "Q 2819 -91 2034 -91 \n",
       "Q 1250 -91 836 529 \n",
       "Q 422 1150 422 2328 \n",
       "Q 422 3509 836 4129 \n",
       "Q 1250 4750 2034 4750 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       </defs>\n",
       "       <use xlink:href=\"#DejaVuSans-2212\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-32\" x=\"83.789062\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"147.412109\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_2\">\n",
       "     <g id=\"line2d_3\">\n",
       "      <path d=\"M 146.632812 143.1 \n",
       "L 146.632812 7.2 \n",
       "\" clip-path=\"url(#pc9dd1287a9)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_4\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#mf2d922344a\" x=\"146.632812\" y=\"143.1\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_2\">\n",
       "      <!-- 0 -->\n",
       "      <g transform=\"translate(143.451562 157.698438)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#DejaVuSans-30\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_3\">\n",
       "     <g id=\"line2d_5\">\n",
       "      <path d=\"M 210.536875 143.1 \n",
       "L 210.536875 7.2 \n",
       "\" clip-path=\"url(#pc9dd1287a9)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_6\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#mf2d922344a\" x=\"210.536875\" y=\"143.1\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_3\">\n",
       "      <!-- 20 -->\n",
       "      <g transform=\"translate(204.174375 157.698438)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#DejaVuSans-32\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"63.623047\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"text_4\">\n",
       "     <!-- x -->\n",
       "     <g transform=\"translate(143.673438 171.376563)scale(0.1 -0.1)\">\n",
       "      <defs>\n",
       "       <path id=\"DejaVuSans-78\" d=\"M 3513 3500 \n",
       "L 2247 1797 \n",
       "L 3578 0 \n",
       "L 2900 0 \n",
       "L 1881 1375 \n",
       "L 863 0 \n",
       "L 184 0 \n",
       "L 1544 1831 \n",
       "L 300 3500 \n",
       "L 978 3500 \n",
       "L 1906 2253 \n",
       "L 2834 3500 \n",
       "L 3513 3500 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "      </defs>\n",
       "      <use xlink:href=\"#DejaVuSans-78\"/>\n",
       "     </g>\n",
       "    </g>\n",
       "   </g>\n",
       "   <g id=\"matplotlib.axis_2\">\n",
       "    <g id=\"ytick_1\">\n",
       "     <g id=\"line2d_7\">\n",
       "      <path d=\"M 48.982813 121.334161 \n",
       "L 244.282813 121.334161 \n",
       "\" clip-path=\"url(#pc9dd1287a9)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_8\">\n",
       "      <defs>\n",
       "       <path id=\"m24b54810d0\" d=\"M 0 0 \n",
       "L -3.5 0 \n",
       "\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </defs>\n",
       "      <g>\n",
       "       <use xlink:href=\"#m24b54810d0\" x=\"48.982813\" y=\"121.334161\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_5\">\n",
       "      <!-- −20 -->\n",
       "      <g transform=\"translate(20.878125 125.133379)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#DejaVuSans-2212\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-32\" x=\"83.789062\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"147.412109\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_2\">\n",
       "     <g id=\"line2d_9\">\n",
       "      <path d=\"M 48.982813 98.242083 \n",
       "L 244.282813 98.242083 \n",
       "\" clip-path=\"url(#pc9dd1287a9)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_10\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#m24b54810d0\" x=\"48.982813\" y=\"98.242083\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_6\">\n",
       "      <!-- −10 -->\n",
       "      <g transform=\"translate(20.878125 102.041301)scale(0.1 -0.1)\">\n",
       "       <defs>\n",
       "        <path id=\"DejaVuSans-31\" d=\"M 794 531 \n",
       "L 1825 531 \n",
       "L 1825 4091 \n",
       "L 703 3866 \n",
       "L 703 4441 \n",
       "L 1819 4666 \n",
       "L 2450 4666 \n",
       "L 2450 531 \n",
       "L 3481 531 \n",
       "L 3481 0 \n",
       "L 794 0 \n",
       "L 794 531 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       </defs>\n",
       "       <use xlink:href=\"#DejaVuSans-2212\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-31\" x=\"83.789062\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"147.412109\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_3\">\n",
       "     <g id=\"line2d_11\">\n",
       "      <path d=\"M 48.982813 75.150004 \n",
       "L 244.282813 75.150004 \n",
       "\" clip-path=\"url(#pc9dd1287a9)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_12\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#m24b54810d0\" x=\"48.982813\" y=\"75.150004\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_7\">\n",
       "      <!-- 0 -->\n",
       "      <g transform=\"translate(35.620313 78.949223)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#DejaVuSans-30\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_4\">\n",
       "     <g id=\"line2d_13\">\n",
       "      <path d=\"M 48.982813 52.057926 \n",
       "L 244.282813 52.057926 \n",
       "\" clip-path=\"url(#pc9dd1287a9)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_14\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#m24b54810d0\" x=\"48.982813\" y=\"52.057926\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_8\">\n",
       "      <!-- 10 -->\n",
       "      <g transform=\"translate(29.257813 55.857145)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#DejaVuSans-31\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"63.623047\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_5\">\n",
       "     <g id=\"line2d_15\">\n",
       "      <path d=\"M 48.982813 28.965848 \n",
       "L 244.282813 28.965848 \n",
       "\" clip-path=\"url(#pc9dd1287a9)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_16\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#m24b54810d0\" x=\"48.982813\" y=\"28.965848\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_9\">\n",
       "      <!-- 20 -->\n",
       "      <g transform=\"translate(29.257813 32.765067)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#DejaVuSans-32\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"63.623047\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"text_10\">\n",
       "     <!-- f(x) -->\n",
       "     <g transform=\"translate(14.798438 83.771094)rotate(-90)scale(0.1 -0.1)\">\n",
       "      <defs>\n",
       "       <path id=\"DejaVuSans-66\" d=\"M 2375 4863 \n",
       "L 2375 4384 \n",
       "L 1825 4384 \n",
       "Q 1516 4384 1395 4259 \n",
       "Q 1275 4134 1275 3809 \n",
       "L 1275 3500 \n",
       "L 2222 3500 \n",
       "L 2222 3053 \n",
       "L 1275 3053 \n",
       "L 1275 0 \n",
       "L 697 0 \n",
       "L 697 3053 \n",
       "L 147 3053 \n",
       "L 147 3500 \n",
       "L 697 3500 \n",
       "L 697 3744 \n",
       "Q 697 4328 969 4595 \n",
       "Q 1241 4863 1831 4863 \n",
       "L 2375 4863 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-28\" d=\"M 1984 4856 \n",
       "Q 1566 4138 1362 3434 \n",
       "Q 1159 2731 1159 2009 \n",
       "Q 1159 1288 1364 580 \n",
       "Q 1569 -128 1984 -844 \n",
       "L 1484 -844 \n",
       "Q 1016 -109 783 600 \n",
       "Q 550 1309 550 2009 \n",
       "Q 550 2706 781 3412 \n",
       "Q 1013 4119 1484 4856 \n",
       "L 1984 4856 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-29\" d=\"M 513 4856 \n",
       "L 1013 4856 \n",
       "Q 1481 4119 1714 3412 \n",
       "Q 1947 2706 1947 2009 \n",
       "Q 1947 1309 1714 600 \n",
       "Q 1481 -109 1013 -844 \n",
       "L 513 -844 \n",
       "Q 928 -128 1133 580 \n",
       "Q 1338 1288 1338 2009 \n",
       "Q 1338 2731 1133 3434 \n",
       "Q 928 4138 513 4856 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "      </defs>\n",
       "      <use xlink:href=\"#DejaVuSans-66\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-28\" x=\"35.205078\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-78\" x=\"74.21875\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-29\" x=\"133.398438\"/>\n",
       "     </g>\n",
       "    </g>\n",
       "   </g>\n",
       "   <g id=\"line2d_17\">\n",
       "    <path d=\"M 57.860085 130.630593 \n",
       "L 58.690839 133.600136 \n",
       "L 59.393781 135.384032 \n",
       "L 60.000872 136.379287 \n",
       "L 60.512104 136.824142 \n",
       "L 60.927477 136.921895 \n",
       "L 61.342857 136.785185 \n",
       "L 61.79019 136.378934 \n",
       "L 62.301422 135.591557 \n",
       "L 62.908506 134.220602 \n",
       "L 63.643401 131.952571 \n",
       "L 64.506107 128.489803 \n",
       "L 65.496619 123.54133 \n",
       "L 66.678847 116.45201 \n",
       "L 68.148643 106.212157 \n",
       "L 70.289423 89.536543 \n",
       "L 74.12367 59.579166 \n",
       "L 75.721271 48.919626 \n",
       "L 77.031304 41.585852 \n",
       "L 78.149626 36.520711 \n",
       "L 79.108185 33.14269 \n",
       "L 79.938938 30.972341 \n",
       "L 80.641887 29.698751 \n",
       "L 81.248977 29.01727 \n",
       "L 81.760209 28.743383 \n",
       "L 82.239488 28.733358 \n",
       "L 82.718768 28.958682 \n",
       "L 83.23 29.453259 \n",
       "L 83.837084 30.37181 \n",
       "L 84.540032 31.866536 \n",
       "L 85.37078 34.194101 \n",
       "L 86.361297 37.694113 \n",
       "L 87.543525 42.767301 \n",
       "L 89.013316 50.150417 \n",
       "L 91.122149 62.039573 \n",
       "L 94.988348 83.923689 \n",
       "L 96.585947 91.624331 \n",
       "L 97.895979 96.915357 \n",
       "L 99.014302 100.569831 \n",
       "L 100.004813 103.082637 \n",
       "L 100.867519 104.692 \n",
       "L 101.602418 105.63016 \n",
       "L 102.241458 106.122308 \n",
       "L 102.816592 106.310654 \n",
       "L 103.39173 106.262399 \n",
       "L 103.966864 105.983805 \n",
       "L 104.605907 105.413848 \n",
       "L 105.340805 104.436356 \n",
       "L 106.171556 102.945478 \n",
       "L 107.162067 100.686479 \n",
       "L 108.344295 97.40669 \n",
       "L 109.846038 92.541074 \n",
       "L 112.242444 83.896549 \n",
       "L 115.213981 73.366242 \n",
       "L 116.843535 68.398749 \n",
       "L 118.18552 64.9901 \n",
       "L 119.335792 62.645275 \n",
       "L 120.358259 61.043905 \n",
       "L 121.252915 60.027156 \n",
       "L 122.083668 59.404774 \n",
       "L 122.850517 59.100057 \n",
       "L 123.585414 59.042627 \n",
       "L 124.320311 59.203493 \n",
       "L 125.119111 59.609444 \n",
       "L 126.013769 60.323785 \n",
       "L 127.068184 61.47173 \n",
       "L 128.346266 63.219121 \n",
       "L 130.103628 66.042762 \n",
       "L 134.417152 73.130377 \n",
       "L 135.886946 75.040979 \n",
       "L 137.133074 76.309805 \n",
       "L 138.251395 77.139628 \n",
       "L 139.273861 77.631716 \n",
       "L 140.264373 77.867359 \n",
       "L 141.254887 77.876908 \n",
       "L 142.309303 77.660658 \n",
       "L 143.491529 77.180935 \n",
       "L 144.92937 76.342454 \n",
       "L 147.613341 74.448763 \n",
       "L 149.498511 73.261171 \n",
       "L 150.808544 72.686589 \n",
       "L 151.926865 72.43211 \n",
       "L 152.94933 72.42667 \n",
       "L 153.939843 72.65011 \n",
       "L 154.930355 73.11038 \n",
       "L 155.952821 73.83659 \n",
       "L 157.071142 74.913786 \n",
       "L 158.317271 76.432515 \n",
       "L 159.819018 78.631182 \n",
       "L 162.023706 82.308106 \n",
       "L 164.963294 87.146223 \n",
       "L 166.369183 89.035444 \n",
       "L 167.487504 90.189224 \n",
       "L 168.446065 90.869772 \n",
       "L 169.276819 91.195271 \n",
       "L 170.043667 91.25703 \n",
       "L 170.778564 91.087316 \n",
       "L 171.513461 90.683763 \n",
       "L 172.28031 90.006537 \n",
       "L 173.143013 88.928355 \n",
       "L 174.069622 87.399934 \n",
       "L 175.124042 85.209395 \n",
       "L 176.306264 82.221877 \n",
       "L 177.712152 78.032974 \n",
       "L 179.501467 71.94848 \n",
       "L 185.093075 52.382355 \n",
       "L 186.371155 48.955419 \n",
       "L 187.425569 46.712774 \n",
       "L 188.320228 45.296142 \n",
       "L 189.087076 44.4752 \n",
       "L 189.726116 44.087495 \n",
       "L 190.301254 43.979388 \n",
       "L 190.844442 44.093122 \n",
       "L 191.419576 44.446838 \n",
       "L 192.058616 45.125017 \n",
       "L 192.761562 46.219401 \n",
       "L 193.560359 47.903235 \n",
       "L 194.455018 50.332755 \n",
       "L 195.477482 53.780638 \n",
       "L 196.659707 58.587519 \n",
       "L 198.065601 65.288813 \n",
       "L 199.88686 75.15378 \n",
       "L 205.574324 106.838877 \n",
       "L 206.916309 112.652587 \n",
       "L 208.002679 116.440748 \n",
       "L 208.929291 118.912859 \n",
       "L 209.696139 120.376643 \n",
       "L 210.335182 121.168029 \n",
       "L 210.87836 121.522082 \n",
       "L 211.357646 121.585043 \n",
       "L 211.836925 121.410173 \n",
       "L 212.348157 120.95868 \n",
       "L 212.923288 120.122039 \n",
       "L 213.594284 118.706717 \n",
       "L 214.361132 116.516725 \n",
       "L 215.223839 113.34237 \n",
       "L 216.214356 108.814651 \n",
       "L 217.364625 102.468655 \n",
       "L 218.738563 93.565332 \n",
       "L 220.495923 80.585678 \n",
       "L 226.79047 32.544559 \n",
       "L 228.068557 25.15944 \n",
       "L 229.122973 20.250096 \n",
       "L 230.017632 17.046236 \n",
       "L 230.78448 15.06564 \n",
       "L 231.423517 13.985428 \n",
       "L 231.934749 13.507618 \n",
       "L 232.382076 13.377326 \n",
       "L 232.797456 13.500003 \n",
       "L 233.244782 13.896603 \n",
       "L 233.75602 14.686958 \n",
       "L 234.363111 16.091744 \n",
       "L 235.066053 18.344575 \n",
       "L 235.385569 19.587277 \n",
       "L 235.385569 19.587277 \n",
       "\" clip-path=\"url(#pc9dd1287a9)\" style=\"fill: none; stroke: #1f77b4; stroke-width: 1.5; stroke-linecap: square\"/>\n",
       "   </g>\n",
       "   <g id=\"line2d_18\">\n",
       "    <path d=\"M 178.584844 75.150004 \n",
       "L 162.608828 83.314287 \n",
       "L 226.496497 34.435752 \n",
       "L 235.40554 19.669416 \n",
       "L 232.264564 13.385311 \n",
       "L 232.373458 13.377273 \n",
       "L 232.373312 13.377273 \n",
       "L 232.373318 13.377273 \n",
       "L 232.373318 13.377273 \n",
       "L 232.373318 13.377273 \n",
       "L 232.373318 13.377273 \n",
       "\" clip-path=\"url(#pc9dd1287a9)\" style=\"fill: none; stroke: #ff7f0e; stroke-width: 1.5; stroke-linecap: square\"/>\n",
       "    <defs>\n",
       "     <path id=\"m5feaa7c4a8\" d=\"M 0 3 \n",
       "C 0.795609 3 1.55874 2.683901 2.12132 2.12132 \n",
       "C 2.683901 1.55874 3 0.795609 3 0 \n",
       "C 3 -0.795609 2.683901 -1.55874 2.12132 -2.12132 \n",
       "C 1.55874 -2.683901 0.795609 -3 0 -3 \n",
       "C -0.795609 -3 -1.55874 -2.683901 -2.12132 -2.12132 \n",
       "C -2.683901 -1.55874 -3 -0.795609 -3 0 \n",
       "C -3 0.795609 -2.683901 1.55874 -2.12132 2.12132 \n",
       "C -1.55874 2.683901 -0.795609 3 0 3 \n",
       "z\n",
       "\" style=\"stroke: #ff7f0e\"/>\n",
       "    </defs>\n",
       "    <g clip-path=\"url(#pc9dd1287a9)\">\n",
       "     <use xlink:href=\"#m5feaa7c4a8\" x=\"178.584844\" y=\"75.150004\" style=\"fill: #ff7f0e; stroke: #ff7f0e\"/>\n",
       "     <use xlink:href=\"#m5feaa7c4a8\" x=\"162.608828\" y=\"83.314287\" style=\"fill: #ff7f0e; stroke: #ff7f0e\"/>\n",
       "     <use xlink:href=\"#m5feaa7c4a8\" x=\"226.496497\" y=\"34.435752\" style=\"fill: #ff7f0e; stroke: #ff7f0e\"/>\n",
       "     <use xlink:href=\"#m5feaa7c4a8\" x=\"235.40554\" y=\"19.669416\" style=\"fill: #ff7f0e; stroke: #ff7f0e\"/>\n",
       "     <use xlink:href=\"#m5feaa7c4a8\" x=\"232.264564\" y=\"13.385311\" style=\"fill: #ff7f0e; stroke: #ff7f0e\"/>\n",
       "     <use xlink:href=\"#m5feaa7c4a8\" x=\"232.373458\" y=\"13.377273\" style=\"fill: #ff7f0e; stroke: #ff7f0e\"/>\n",
       "     <use xlink:href=\"#m5feaa7c4a8\" x=\"232.373312\" y=\"13.377273\" style=\"fill: #ff7f0e; stroke: #ff7f0e\"/>\n",
       "     <use xlink:href=\"#m5feaa7c4a8\" x=\"232.373318\" y=\"13.377273\" style=\"fill: #ff7f0e; stroke: #ff7f0e\"/>\n",
       "     <use xlink:href=\"#m5feaa7c4a8\" x=\"232.373318\" y=\"13.377273\" style=\"fill: #ff7f0e; stroke: #ff7f0e\"/>\n",
       "     <use xlink:href=\"#m5feaa7c4a8\" x=\"232.373318\" y=\"13.377273\" style=\"fill: #ff7f0e; stroke: #ff7f0e\"/>\n",
       "     <use xlink:href=\"#m5feaa7c4a8\" x=\"232.373318\" y=\"13.377273\" style=\"fill: #ff7f0e; stroke: #ff7f0e\"/>\n",
       "    </g>\n",
       "   </g>\n",
       "   <g id=\"patch_3\">\n",
       "    <path d=\"M 48.982813 143.1 \n",
       "L 48.982813 7.2 \n",
       "\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_4\">\n",
       "    <path d=\"M 244.282813 143.1 \n",
       "L 244.282813 7.2 \n",
       "\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_5\">\n",
       "    <path d=\"M 48.982813 143.1 \n",
       "L 244.282812 143.1 \n",
       "\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_6\">\n",
       "    <path d=\"M 48.982813 7.2 \n",
       "L 244.282812 7.2 \n",
       "\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n",
       "   </g>\n",
       "  </g>\n",
       " </g>\n",
       " <defs>\n",
       "  <clipPath id=\"pc9dd1287a9\">\n",
       "   <rect x=\"48.982813\" y=\"7.2\" width=\"195.3\" height=\"135.9\"/>\n",
       "  </clipPath>\n",
       " </defs>\n",
       "</svg>\n"
      ],
      "text/plain": [
       "<Figure size 252x180 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "c = torch.tensor(0.15 * np.pi)\n",
    "\n",
    "def f(x):  # 目标函数\n",
    "    return x * torch.cos(c * x)\n",
    "\n",
    "def f_grad(x):  # 目标函数的梯度\n",
    "    return torch.cos(c * x) - c * x * torch.sin(c * x)\n",
    "\n",
    "def f_hess(x):  # 目标函数的Hessian\n",
    "    return - 2 * c * torch.sin(c * x) - x * c**2 * torch.cos(c * x)\n",
    "\n",
    "show_trace(newton(), f)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "28696853",
   "metadata": {
    "origin_pos": 29
   },
   "source": [
    "这发生了惊人的错误。我们怎样才能修正它？\n",
    "一种方法是用取Hessian的绝对值来修正，另一个策略是重新引入学习率。\n",
    "这似乎违背了初衷，但不完全是——拥有二阶信息可以使我们在曲率较大时保持谨慎，而在目标函数较平坦时则采用较大的学习率。\n",
    "让我们看看在学习率稍小的情况下它是如何生效的，比如$\\eta = 0.5$。\n",
    "如我们所见，我们有了一个相当高效的算法。\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "1d0aa6d6",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-08-18T07:01:28.666394Z",
     "iopub.status.busy": "2023-08-18T07:01:28.665576Z",
     "iopub.status.idle": "2023-08-18T07:01:28.904465Z",
     "shell.execute_reply": "2023-08-18T07:01:28.903311Z"
    },
    "origin_pos": 30,
    "tab": [
     "pytorch"
    ]
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "epoch 10, x: tensor(7.2699)\n"
     ]
    },
    {
     "data": {
      "image/svg+xml": [
       "<?xml version=\"1.0\" encoding=\"utf-8\" standalone=\"no\"?>\n",
       "<!DOCTYPE svg PUBLIC \"-//W3C//DTD SVG 1.1//EN\"\n",
       "  \"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd\">\n",
       "<svg xmlns:xlink=\"http://www.w3.org/1999/xlink\" width=\"245.120313pt\" height=\"180.65625pt\" viewBox=\"0 0 245.120313 180.65625\" xmlns=\"http://www.w3.org/2000/svg\" version=\"1.1\">\n",
       " <metadata>\n",
       "  <rdf:RDF xmlns:dc=\"http://purl.org/dc/elements/1.1/\" xmlns:cc=\"http://creativecommons.org/ns#\" xmlns:rdf=\"http://www.w3.org/1999/02/22-rdf-syntax-ns#\">\n",
       "   <cc:Work>\n",
       "    <dc:type rdf:resource=\"http://purl.org/dc/dcmitype/StillImage\"/>\n",
       "    <dc:date>2023-08-18T07:01:28.869577</dc:date>\n",
       "    <dc:format>image/svg+xml</dc:format>\n",
       "    <dc:creator>\n",
       "     <cc:Agent>\n",
       "      <dc:title>Matplotlib v3.5.1, https://matplotlib.org/</dc:title>\n",
       "     </cc:Agent>\n",
       "    </dc:creator>\n",
       "   </cc:Work>\n",
       "  </rdf:RDF>\n",
       " </metadata>\n",
       " <defs>\n",
       "  <style type=\"text/css\">*{stroke-linejoin: round; stroke-linecap: butt}</style>\n",
       " </defs>\n",
       " <g id=\"figure_1\">\n",
       "  <g id=\"patch_1\">\n",
       "   <path d=\"M 0 180.65625 \n",
       "L 245.120313 180.65625 \n",
       "L 245.120313 0 \n",
       "L 0 0 \n",
       "L 0 180.65625 \n",
       "z\n",
       "\" style=\"fill: none\"/>\n",
       "  </g>\n",
       "  <g id=\"axes_1\">\n",
       "   <g id=\"patch_2\">\n",
       "    <path d=\"M 42.620312 143.1 \n",
       "L 237.920313 143.1 \n",
       "L 237.920313 7.2 \n",
       "L 42.620312 7.2 \n",
       "z\n",
       "\" style=\"fill: #ffffff\"/>\n",
       "   </g>\n",
       "   <g id=\"matplotlib.axis_1\">\n",
       "    <g id=\"xtick_1\">\n",
       "     <g id=\"line2d_1\">\n",
       "      <path d=\"M 51.497585 143.1 \n",
       "L 51.497585 7.2 \n",
       "\" clip-path=\"url(#p427918317c)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_2\">\n",
       "      <defs>\n",
       "       <path id=\"m5e67eabe5f\" d=\"M 0 0 \n",
       "L 0 3.5 \n",
       "\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </defs>\n",
       "      <g>\n",
       "       <use xlink:href=\"#m5e67eabe5f\" x=\"51.497585\" y=\"143.1\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_1\">\n",
       "      <!-- −10 -->\n",
       "      <g transform=\"translate(40.945241 157.698438)scale(0.1 -0.1)\">\n",
       "       <defs>\n",
       "        <path id=\"DejaVuSans-2212\" d=\"M 678 2272 \n",
       "L 4684 2272 \n",
       "L 4684 1741 \n",
       "L 678 1741 \n",
       "L 678 2272 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "        <path id=\"DejaVuSans-31\" d=\"M 794 531 \n",
       "L 1825 531 \n",
       "L 1825 4091 \n",
       "L 703 3866 \n",
       "L 703 4441 \n",
       "L 1819 4666 \n",
       "L 2450 4666 \n",
       "L 2450 531 \n",
       "L 3481 531 \n",
       "L 3481 0 \n",
       "L 794 0 \n",
       "L 794 531 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "        <path id=\"DejaVuSans-30\" d=\"M 2034 4250 \n",
       "Q 1547 4250 1301 3770 \n",
       "Q 1056 3291 1056 2328 \n",
       "Q 1056 1369 1301 889 \n",
       "Q 1547 409 2034 409 \n",
       "Q 2525 409 2770 889 \n",
       "Q 3016 1369 3016 2328 \n",
       "Q 3016 3291 2770 3770 \n",
       "Q 2525 4250 2034 4250 \n",
       "z\n",
       "M 2034 4750 \n",
       "Q 2819 4750 3233 4129 \n",
       "Q 3647 3509 3647 2328 \n",
       "Q 3647 1150 3233 529 \n",
       "Q 2819 -91 2034 -91 \n",
       "Q 1250 -91 836 529 \n",
       "Q 422 1150 422 2328 \n",
       "Q 422 3509 836 4129 \n",
       "Q 1250 4750 2034 4750 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       </defs>\n",
       "       <use xlink:href=\"#DejaVuSans-2212\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-31\" x=\"83.789062\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"147.412109\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_2\">\n",
       "     <g id=\"line2d_3\">\n",
       "      <path d=\"M 95.883949 143.1 \n",
       "L 95.883949 7.2 \n",
       "\" clip-path=\"url(#p427918317c)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_4\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#m5e67eabe5f\" x=\"95.883949\" y=\"143.1\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_2\">\n",
       "      <!-- −5 -->\n",
       "      <g transform=\"translate(88.512855 157.698438)scale(0.1 -0.1)\">\n",
       "       <defs>\n",
       "        <path id=\"DejaVuSans-35\" d=\"M 691 4666 \n",
       "L 3169 4666 \n",
       "L 3169 4134 \n",
       "L 1269 4134 \n",
       "L 1269 2991 \n",
       "Q 1406 3038 1543 3061 \n",
       "Q 1681 3084 1819 3084 \n",
       "Q 2600 3084 3056 2656 \n",
       "Q 3513 2228 3513 1497 \n",
       "Q 3513 744 3044 326 \n",
       "Q 2575 -91 1722 -91 \n",
       "Q 1428 -91 1123 -41 \n",
       "Q 819 9 494 109 \n",
       "L 494 744 \n",
       "Q 775 591 1075 516 \n",
       "Q 1375 441 1709 441 \n",
       "Q 2250 441 2565 725 \n",
       "Q 2881 1009 2881 1497 \n",
       "Q 2881 1984 2565 2268 \n",
       "Q 2250 2553 1709 2553 \n",
       "Q 1456 2553 1204 2497 \n",
       "Q 953 2441 691 2322 \n",
       "L 691 4666 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       </defs>\n",
       "       <use xlink:href=\"#DejaVuSans-2212\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-35\" x=\"83.789062\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_3\">\n",
       "     <g id=\"line2d_5\">\n",
       "      <path d=\"M 140.270312 143.1 \n",
       "L 140.270312 7.2 \n",
       "\" clip-path=\"url(#p427918317c)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_6\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#m5e67eabe5f\" x=\"140.270312\" y=\"143.1\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_3\">\n",
       "      <!-- 0 -->\n",
       "      <g transform=\"translate(137.089062 157.698438)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#DejaVuSans-30\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_4\">\n",
       "     <g id=\"line2d_7\">\n",
       "      <path d=\"M 184.656676 143.1 \n",
       "L 184.656676 7.2 \n",
       "\" clip-path=\"url(#p427918317c)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_8\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#m5e67eabe5f\" x=\"184.656676\" y=\"143.1\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_4\">\n",
       "      <!-- 5 -->\n",
       "      <g transform=\"translate(181.475426 157.698438)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#DejaVuSans-35\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_5\">\n",
       "     <g id=\"line2d_9\">\n",
       "      <path d=\"M 229.04304 143.1 \n",
       "L 229.04304 7.2 \n",
       "\" clip-path=\"url(#p427918317c)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_10\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#m5e67eabe5f\" x=\"229.04304\" y=\"143.1\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_5\">\n",
       "      <!-- 10 -->\n",
       "      <g transform=\"translate(222.68054 157.698438)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#DejaVuSans-31\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"63.623047\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"text_6\">\n",
       "     <!-- x -->\n",
       "     <g transform=\"translate(137.310937 171.376563)scale(0.1 -0.1)\">\n",
       "      <defs>\n",
       "       <path id=\"DejaVuSans-78\" d=\"M 3513 3500 \n",
       "L 2247 1797 \n",
       "L 3578 0 \n",
       "L 2900 0 \n",
       "L 1881 1375 \n",
       "L 863 0 \n",
       "L 184 0 \n",
       "L 1544 1831 \n",
       "L 300 3500 \n",
       "L 978 3500 \n",
       "L 1906 2253 \n",
       "L 2834 3500 \n",
       "L 3513 3500 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "      </defs>\n",
       "      <use xlink:href=\"#DejaVuSans-78\"/>\n",
       "     </g>\n",
       "    </g>\n",
       "   </g>\n",
       "   <g id=\"matplotlib.axis_2\">\n",
       "    <g id=\"ytick_1\">\n",
       "     <g id=\"line2d_11\">\n",
       "      <path d=\"M 42.620312 119.411597 \n",
       "L 237.920313 119.411597 \n",
       "\" clip-path=\"url(#p427918317c)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_12\">\n",
       "      <defs>\n",
       "       <path id=\"m4fe7b34166\" d=\"M 0 0 \n",
       "L -3.5 0 \n",
       "\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </defs>\n",
       "      <g>\n",
       "       <use xlink:href=\"#m4fe7b34166\" x=\"42.620312\" y=\"119.411597\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_7\">\n",
       "      <!-- −5 -->\n",
       "      <g transform=\"translate(20.878125 123.210816)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#DejaVuSans-2212\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-35\" x=\"83.789062\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_2\">\n",
       "     <g id=\"line2d_13\">\n",
       "      <path d=\"M 42.620312 75.15 \n",
       "L 237.920313 75.15 \n",
       "\" clip-path=\"url(#p427918317c)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_14\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#m4fe7b34166\" x=\"42.620312\" y=\"75.15\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_8\">\n",
       "      <!-- 0 -->\n",
       "      <g transform=\"translate(29.257812 78.949219)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#DejaVuSans-30\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_3\">\n",
       "     <g id=\"line2d_15\">\n",
       "      <path d=\"M 42.620312 30.888403 \n",
       "L 237.920313 30.888403 \n",
       "\" clip-path=\"url(#p427918317c)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_16\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#m4fe7b34166\" x=\"42.620312\" y=\"30.888403\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_9\">\n",
       "      <!-- 5 -->\n",
       "      <g transform=\"translate(29.257812 34.687622)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#DejaVuSans-35\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"text_10\">\n",
       "     <!-- f(x) -->\n",
       "     <g transform=\"translate(14.798437 83.771094)rotate(-90)scale(0.1 -0.1)\">\n",
       "      <defs>\n",
       "       <path id=\"DejaVuSans-66\" d=\"M 2375 4863 \n",
       "L 2375 4384 \n",
       "L 1825 4384 \n",
       "Q 1516 4384 1395 4259 \n",
       "Q 1275 4134 1275 3809 \n",
       "L 1275 3500 \n",
       "L 2222 3500 \n",
       "L 2222 3053 \n",
       "L 1275 3053 \n",
       "L 1275 0 \n",
       "L 697 0 \n",
       "L 697 3053 \n",
       "L 147 3053 \n",
       "L 147 3500 \n",
       "L 697 3500 \n",
       "L 697 3744 \n",
       "Q 697 4328 969 4595 \n",
       "Q 1241 4863 1831 4863 \n",
       "L 2375 4863 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-28\" d=\"M 1984 4856 \n",
       "Q 1566 4138 1362 3434 \n",
       "Q 1159 2731 1159 2009 \n",
       "Q 1159 1288 1364 580 \n",
       "Q 1569 -128 1984 -844 \n",
       "L 1484 -844 \n",
       "Q 1016 -109 783 600 \n",
       "Q 550 1309 550 2009 \n",
       "Q 550 2706 781 3412 \n",
       "Q 1013 4119 1484 4856 \n",
       "L 1984 4856 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-29\" d=\"M 513 4856 \n",
       "L 1013 4856 \n",
       "Q 1481 4119 1714 3412 \n",
       "Q 1947 2706 1947 2009 \n",
       "Q 1947 1309 1714 600 \n",
       "Q 1481 -109 1013 -844 \n",
       "L 513 -844 \n",
       "Q 928 -128 1133 580 \n",
       "Q 1338 1288 1338 2009 \n",
       "Q 1338 2731 1133 3434 \n",
       "Q 928 4138 513 4856 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "      </defs>\n",
       "      <use xlink:href=\"#DejaVuSans-66\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-28\" x=\"35.205078\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-78\" x=\"74.21875\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-29\" x=\"133.398438\"/>\n",
       "     </g>\n",
       "    </g>\n",
       "   </g>\n",
       "   <g id=\"line2d_17\">\n",
       "    <path d=\"M 51.497585 75.150001 \n",
       "L 54.515859 61.507494 \n",
       "L 57.090268 50.885046 \n",
       "L 59.309586 42.62211 \n",
       "L 61.351355 35.837137 \n",
       "L 63.215583 30.377167 \n",
       "L 64.991044 25.861634 \n",
       "L 66.588956 22.386195 \n",
       "L 68.098084 19.624675 \n",
       "L 69.518451 17.490893 \n",
       "L 70.850038 15.899777 \n",
       "L 72.092859 14.769329 \n",
       "L 73.246906 14.022221 \n",
       "L 74.312178 13.586851 \n",
       "L 75.377449 13.390552 \n",
       "L 76.442725 13.427546 \n",
       "L 77.507997 13.691167 \n",
       "L 78.662039 14.22383 \n",
       "L 79.816086 15.003166 \n",
       "L 81.058907 16.105109 \n",
       "L 82.479269 17.678382 \n",
       "L 83.988402 19.689963 \n",
       "L 85.675089 22.313917 \n",
       "L 87.628087 25.787156 \n",
       "L 89.847405 30.206604 \n",
       "L 92.599359 36.214766 \n",
       "L 96.59413 45.535707 \n",
       "L 102.719453 59.79435 \n",
       "L 105.648948 66.04003 \n",
       "L 108.134584 70.834186 \n",
       "L 110.353902 74.634558 \n",
       "L 112.306903 77.556075 \n",
       "L 114.17113 79.946414 \n",
       "L 115.857812 81.757913 \n",
       "L 117.45572 83.158273 \n",
       "L 118.964857 84.195921 \n",
       "L 120.385222 84.920387 \n",
       "L 121.805586 85.403652 \n",
       "L 123.225949 85.651421 \n",
       "L 124.646313 85.67174 \n",
       "L 126.066676 85.474901 \n",
       "L 127.575812 85.041746 \n",
       "L 129.173722 84.350543 \n",
       "L 130.949177 83.330131 \n",
       "L 132.902176 81.942544 \n",
       "L 135.210267 80.014888 \n",
       "L 138.22854 77.174086 \n",
       "L 146.04054 69.663818 \n",
       "L 148.34863 67.823794 \n",
       "L 150.301631 66.531906 \n",
       "L 152.077085 65.614112 \n",
       "L 153.674995 65.026673 \n",
       "L 155.18413 64.700263 \n",
       "L 156.604495 64.61058 \n",
       "L 158.024858 64.743475 \n",
       "L 159.445222 65.10813 \n",
       "L 160.865585 65.711433 \n",
       "L 162.285949 66.557882 \n",
       "L 163.795086 67.725879 \n",
       "L 165.392994 69.263456 \n",
       "L 167.079676 71.216922 \n",
       "L 168.855131 73.628385 \n",
       "L 170.808131 76.680042 \n",
       "L 172.938677 80.448158 \n",
       "L 175.33554 85.168809 \n",
       "L 178.087492 91.099238 \n",
       "L 181.815947 99.711378 \n",
       "L 189.272858 117.052872 \n",
       "L 191.936041 122.62392 \n",
       "L 194.155359 126.773504 \n",
       "L 196.019583 129.822238 \n",
       "L 197.706269 132.178722 \n",
       "L 199.215402 133.924756 \n",
       "L 200.635769 135.226991 \n",
       "L 201.878586 136.07617 \n",
       "L 203.032632 136.608833 \n",
       "L 204.097904 136.872454 \n",
       "L 205.16318 136.909448 \n",
       "L 206.228452 136.713149 \n",
       "L 207.293723 136.277775 \n",
       "L 208.358995 135.598428 \n",
       "L 209.513041 134.582583 \n",
       "L 210.667088 133.272041 \n",
       "L 211.909901 131.528413 \n",
       "L 213.241497 129.276672 \n",
       "L 214.661859 126.438757 \n",
       "L 216.259771 122.713826 \n",
       "L 217.946449 118.183227 \n",
       "L 219.721902 112.772058 \n",
       "L 221.674904 106.096761 \n",
       "L 223.805448 98.012828 \n",
       "L 226.202315 88.022895 \n",
       "L 228.954265 75.566753 \n",
       "L 228.954265 75.566753 \n",
       "\" clip-path=\"url(#p427918317c)\" style=\"fill: none; stroke: #1f77b4; stroke-width: 1.5; stroke-linecap: square\"/>\n",
       "   </g>\n",
       "   <g id=\"line2d_18\">\n",
       "    <path d=\"M 229.04304 75.149999 \n",
       "L 206.849858 136.488573 \n",
       "L 205.848192 136.810382 \n",
       "L 205.331865 136.89407 \n",
       "L 205.069008 136.915488 \n",
       "L 204.936286 136.920908 \n",
       "L 204.869587 136.922276 \n",
       "L 204.83615 136.922613 \n",
       "L 204.819408 136.922702 \n",
       "L 204.811031 136.922723 \n",
       "L 204.806841 136.922727 \n",
       "\" clip-path=\"url(#p427918317c)\" style=\"fill: none; stroke: #ff7f0e; stroke-width: 1.5; stroke-linecap: square\"/>\n",
       "    <defs>\n",
       "     <path id=\"m2f04cbd60e\" d=\"M 0 3 \n",
       "C 0.795609 3 1.55874 2.683901 2.12132 2.12132 \n",
       "C 2.683901 1.55874 3 0.795609 3 0 \n",
       "C 3 -0.795609 2.683901 -1.55874 2.12132 -2.12132 \n",
       "C 1.55874 -2.683901 0.795609 -3 0 -3 \n",
       "C -0.795609 -3 -1.55874 -2.683901 -2.12132 -2.12132 \n",
       "C -2.683901 -1.55874 -3 -0.795609 -3 0 \n",
       "C -3 0.795609 -2.683901 1.55874 -2.12132 2.12132 \n",
       "C -1.55874 2.683901 -0.795609 3 0 3 \n",
       "z\n",
       "\" style=\"stroke: #ff7f0e\"/>\n",
       "    </defs>\n",
       "    <g clip-path=\"url(#p427918317c)\">\n",
       "     <use xlink:href=\"#m2f04cbd60e\" x=\"229.04304\" y=\"75.149999\" style=\"fill: #ff7f0e; stroke: #ff7f0e\"/>\n",
       "     <use xlink:href=\"#m2f04cbd60e\" x=\"206.849858\" y=\"136.488573\" style=\"fill: #ff7f0e; stroke: #ff7f0e\"/>\n",
       "     <use xlink:href=\"#m2f04cbd60e\" x=\"205.848192\" y=\"136.810382\" style=\"fill: #ff7f0e; stroke: #ff7f0e\"/>\n",
       "     <use xlink:href=\"#m2f04cbd60e\" x=\"205.331865\" y=\"136.89407\" style=\"fill: #ff7f0e; stroke: #ff7f0e\"/>\n",
       "     <use xlink:href=\"#m2f04cbd60e\" x=\"205.069008\" y=\"136.915488\" style=\"fill: #ff7f0e; stroke: #ff7f0e\"/>\n",
       "     <use xlink:href=\"#m2f04cbd60e\" x=\"204.936286\" y=\"136.920908\" style=\"fill: #ff7f0e; stroke: #ff7f0e\"/>\n",
       "     <use xlink:href=\"#m2f04cbd60e\" x=\"204.869587\" y=\"136.922276\" style=\"fill: #ff7f0e; stroke: #ff7f0e\"/>\n",
       "     <use xlink:href=\"#m2f04cbd60e\" x=\"204.83615\" y=\"136.922613\" style=\"fill: #ff7f0e; stroke: #ff7f0e\"/>\n",
       "     <use xlink:href=\"#m2f04cbd60e\" x=\"204.819408\" y=\"136.922702\" style=\"fill: #ff7f0e; stroke: #ff7f0e\"/>\n",
       "     <use xlink:href=\"#m2f04cbd60e\" x=\"204.811031\" y=\"136.922723\" style=\"fill: #ff7f0e; stroke: #ff7f0e\"/>\n",
       "     <use xlink:href=\"#m2f04cbd60e\" x=\"204.806841\" y=\"136.922727\" style=\"fill: #ff7f0e; stroke: #ff7f0e\"/>\n",
       "    </g>\n",
       "   </g>\n",
       "   <g id=\"patch_3\">\n",
       "    <path d=\"M 42.620312 143.1 \n",
       "L 42.620312 7.2 \n",
       "\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_4\">\n",
       "    <path d=\"M 237.920313 143.1 \n",
       "L 237.920313 7.2 \n",
       "\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_5\">\n",
       "    <path d=\"M 42.620312 143.1 \n",
       "L 237.920313 143.1 \n",
       "\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_6\">\n",
       "    <path d=\"M 42.620312 7.2 \n",
       "L 237.920313 7.2 \n",
       "\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n",
       "   </g>\n",
       "  </g>\n",
       " </g>\n",
       " <defs>\n",
       "  <clipPath id=\"p427918317c\">\n",
       "   <rect x=\"42.620312\" y=\"7.2\" width=\"195.3\" height=\"135.9\"/>\n",
       "  </clipPath>\n",
       " </defs>\n",
       "</svg>\n"
      ],
      "text/plain": [
       "<Figure size 252x180 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "show_trace(newton(0.5), f)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1575b769",
   "metadata": {
    "origin_pos": 31
   },
   "source": [
    "### 收敛性分析\n",
    "\n",
    "在此，我们以部分目标凸函数$f$为例，分析它们的牛顿法收敛速度。\n",
    "这些目标凸函数三次可微，而且二阶导数不为零，即$f'' > 0$。\n",
    "由于多变量情况下的证明是对以下一维参数情况证明的直接拓展，对我们理解这个问题不能提供更多帮助，因此我们省略了多变量情况的证明。\n",
    "\n",
    "用$x^{(k)}$表示$x$在第$k^\\mathrm{th}$次迭代时的值，\n",
    "令$e^{(k)} \\stackrel{\\mathrm{def}}{=} x^{(k)} - x^*$表示$k^\\mathrm{th}$迭代时与最优性的距离。\n",
    "通过泰勒展开，我们得到条件$f'(x^*) = 0$可以写成\n",
    "\n",
    "$$0 = f'(x^{(k)} - e^{(k)}) = f'(x^{(k)}) - e^{(k)} f''(x^{(k)}) + \\frac{1}{2} (e^{(k)})^2 f'''(\\xi^{(k)}),$$\n",
    "\n",
    "这对某些$\\xi^{(k)} \\in [x^{(k)} - e^{(k)}, x^{(k)}]$成立。\n",
    "将上述展开除以$f''(x^{(k)})$得到\n",
    "\n",
    "$$e^{(k)} - \\frac{f'(x^{(k)})}{f''(x^{(k)})} = \\frac{1}{2} (e^{(k)})^2 \\frac{f'''(\\xi^{(k)})}{f''(x^{(k)})}.$$\n",
    "\n",
    "回想之前的方程$x^{(k+1)} = x^{(k)} - f'(x^{(k)}) / f''(x^{(k)})$。\n",
    "代入这个更新方程，取两边的绝对值，我们得到\n",
    "\n",
    "$$\\left|e^{(k+1)}\\right| = \\frac{1}{2}(e^{(k)})^2 \\frac{\\left|f'''(\\xi^{(k)})\\right|}{f''(x^{(k)})}.$$\n",
    "\n",
    "因此，每当我们处于有界区域$\\left|f'''(\\xi^{(k)})\\right| / (2f''(x^{(k)})) \\leq c$，\n",
    "我们就有一个二次递减误差\n",
    "\n",
    "$$\\left|e^{(k+1)}\\right| \\leq c (e^{(k)})^2.$$\n",
    "\n",
    "另一方面，优化研究人员称之为“线性”收敛，而将$\\left|e^{(k+1)}\\right| \\leq \\alpha \\left|e^{(k)}\\right|$这样的条件称为“恒定”收敛速度。\n",
    "请注意，我们无法估计整体收敛的速度，但是一旦我们接近极小值，收敛将变得非常快。\n",
    "另外，这种分析要求$f$在高阶导数上表现良好，即确保$f$在如何变化它的值方面没有任何“超常”的特性。\n",
    "\n",
    "### 预处理\n",
    "\n",
    "计算和存储完整的Hessian非常昂贵，而改善这个问题的一种方法是“预处理”。\n",
    "它回避了计算整个Hessian，而只计算“对角线”项，即如下的算法更新：\n",
    "\n",
    "$$\\mathbf{x} \\leftarrow \\mathbf{x} - \\eta \\mathrm{diag}(\\mathbf{H})^{-1} \\nabla f(\\mathbf{x}).$$\n",
    "\n",
    "虽然这不如完整的牛顿法精确，但它仍然比不使用要好得多。\n",
    "为什么预处理有效呢？\n",
    "假设一个变量以毫米表示高度，另一个变量以公里表示高度的情况。\n",
    "假设这两种自然尺度都以米为单位，那么我们的参数化就出现了严重的不匹配。\n",
    "幸运的是，使用预处理可以消除这种情况。\n",
    "梯度下降的有效预处理相当于为每个变量选择不同的学习率（矢量$\\mathbf{x}$的坐标）。\n",
    "我们将在后面一节看到，预处理推动了随机梯度下降优化算法的一些创新。\n",
    "\n",
    "### 梯度下降和线搜索\n",
    "\n",
    "梯度下降的一个关键问题是我们可能会超过目标或进展不足，\n",
    "解决这一问题的简单方法是结合使用线搜索和梯度下降。\n",
    "也就是说，我们使用$\\nabla f(\\mathbf{x})$给出的方向，\n",
    "然后进行二分搜索，以确定哪个学习率$\\eta$使$f(\\mathbf{x} - \\eta \\nabla f(\\mathbf{x}))$取最小值。\n",
    "\n",
    "有关分析和证明，此算法收敛迅速（请参见 :cite:`Boyd.Vandenberghe.2004`）。\n",
    "然而，对深度学习而言，这不太可行。\n",
    "因为线搜索的每一步都需要评估整个数据集上的目标函数，实现它的方式太昂贵了。\n",
    "\n",
    "## 小结\n",
    "\n",
    "* 学习率的大小很重要：学习率太大会使模型发散，学习率太小会没有进展。\n",
    "* 梯度下降会可能陷入局部极小值，而得不到全局最小值。\n",
    "* 在高维模型中，调整学习率是很复杂的。\n",
    "* 预处理有助于调节比例。\n",
    "* 牛顿法在凸问题中一旦开始正常工作，速度就会快得多。\n",
    "* 对于非凸问题，不要不作任何调整就使用牛顿法。\n",
    "\n",
    "## 练习\n",
    "\n",
    "1. 用不同的学习率和目标函数进行梯度下降实验。\n",
    "1. 在区间$[a, b]$中实现线搜索以最小化凸函数。\n",
    "    1. 是否需要导数来进行二分搜索，即决定选择$[a, (a+b)/2]$还是$[(a+b)/2, b]$。\n",
    "    1. 算法的收敛速度有多快？\n",
    "    1. 实现该算法，并将其应用于求$\\log (\\exp(x) + \\exp(-2x -3))$的最小值。\n",
    "1. 设计一个定义在$\\mathbb{R}^2$上的目标函数，它的梯度下降非常缓慢。提示：不同坐标的缩放方式不同。\n",
    "1. 使用预处理实现牛顿方法的轻量版本。\n",
    "    1. 使用对角Hessian作为预条件子。\n",
    "    1. 使用它的绝对值，而不是实际值（可能有符号）。\n",
    "    1. 将此应用于上述问题。\n",
    "1. 将上述算法应用于多个目标函数（凸或非凸）。如果把坐标旋转$45$度会怎么样？\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5db14585",
   "metadata": {
    "origin_pos": 33,
    "tab": [
     "pytorch"
    ]
   },
   "source": [
    "[Discussions](https://discuss.d2l.ai/t/3836)\n"
   ]
  }
 ],
 "metadata": {
  "language_info": {
   "name": "python"
  },
  "required_libs": []
 },
 "nbformat": 4,
 "nbformat_minor": 5
}